Skip to content

Gateway readiness rejects tagged OpenShell v0.0.101 process identity #8755

Closed
Bug
#8757
Closed
Gateway readiness rejects tagged OpenShell v0.0.101 process identity#8755
Bug
#8757
Assignees
senthilr-nv
Labels
needs: triageAwaiting maintainer classification

Description

@senthilr-nv
senthilr-nv
opened on Aug 10, 2026

Investigation summary

  • Reproduced on CPU-only Ubuntu with NemoClaw cae757ed... and OpenShell v0.0.101; latest main at bafcad8480... has the same readiness implementation.
  • NemoClaw started the expected v0.0.101 gateway, which returned HTTP 200 and listened on both expected addresses.
  • Readiness still marked gateway.version.compatible and gateway.port.uncontested unknown because the trusted-binary check rejected OpenShell's tagged process identity.
  • The target matcher accepted that same identity, and /proc/<pid>/exe resolved to the expected gateway binary.
  • This blocks onboarding before Model Context Protocol (MCP) configuration and is distinct from Investigate trusted-private MCP tool discovery returning zero tools on OpenShell v0.0.101 #8746.

Description

The onboarding readiness check does not recognize a gateway that NemoClaw has just started when OpenShell v0.0.101 uses this tagged process identity:

openshell-gateway[nemoclaw=nemoclaw;port=8080]

The gateway is the expected executable, returns HTTP 200, and owns the expected listeners. The target-bound gateway matcher accepts the tag, but the trusted-binary gate rejects it. Onboarding therefore stops before sandbox or MCP configuration. The retained diagnostic does not record the specific Docker-driver environment-marker values, so this issue does not claim that those values were observed.

Expected result: readiness attributes the tagged listener to the trusted OpenShell gateway and continues onboarding.

Actual result:

System readiness could not confirm required capabilities: gateway.version.compatible, gateway.port.uncontested.
Gateway port 8080 could not be observed completely. Inspect port 8080 and stop only its owning process before retrying.

PR #8738 introduced the affected readiness path after two trusted OpenClaw MCP runs had passed on OpenShell v0.0.101. PR #8751 improves MCP failure diagnostics but does not change this onboarding behavior.

Reproduction steps

  1. Check out NemoClaw cae757edca1e9996dba57d1d6b85f2d0ab1b23bb.

  2. Install the exact OpenShell v0.0.101 CLI.

  3. Place the verified cloudflared 2026.6.1 binary at /home/ubuntu/p014-main-cae757ed/bin/cloudflared, add that directory to PATH, and verify it before the run:

    export PATH="/home/ubuntu/p014-main-cae757ed/bin:${PATH}"
cloudflared --version
printf '%s  %s\n' \
  '5861a10a438fe8ddcfebb3b830f83966cbf193edafce0fe2eeb198fbae1f7a22' \
  '/home/ubuntu/p014-main-cae757ed/bin/cloudflared' | sha256sum --check

  4. Generate the one-run TLS fixture in its owner-only directory and load the three paths written by the repository setup script:

    export RUNNER_TEMP=/home/ubuntu/p014-main-cae757ed/tmp
export GITHUB_ENV="${RUNNER_TEMP}/mcp-tls.env"
install -d -m 700 "${RUNNER_TEMP}"
install -m 600 /dev/null "${GITHUB_ENV}"
bash test/e2e/setup-mcp-test-tls.sh
chmod 600 \
  "${RUNNER_TEMP}/nemoclaw-mcp-tls/ca.key" \
  "${RUNNER_TEMP}/nemoclaw-mcp-tls/server.key"
set -a
source "${GITHUB_ENV}"
set +a

    This exports NEMOCLAW_MCP_TLS_CA_CERT, NEMOCLAW_MCP_TLS_CERT, and NEMOCLAW_MCP_TLS_KEY. Remove ${RUNNER_TEMP} after the run; this deletes both private keys and the environment file.

  5. On the CPU-only Ubuntu 24.04 host, run the OpenClaw MCP live E2E in full-lifecycle mode:

    NEMOCLAW_MCP_BRIDGE_AGENT=openclaw \
  NEMOCLAW_OPENSHELL_EXACT_MAIN_PROOF=1 \
  NEMOCLAW_RUN_LIVE_E2E=1 \
  npx tsx tools/e2e/live-vitest-invocation.mts run \
    --test-path test/e2e/live/mcp-bridge.test.ts

  6. Phase 1 starts the compatible inference and MCP endpoints.

  7. During phase 2, observe that the gateway starts and returns HTTP 200, but onboarding exits with the readiness error above.

  8. Inspect the gateway process and listeners. The observed process identity is openshell-gateway[nemoclaw=nemoclaw;port=8080], while /proc/<pid>/exe resolves to the expected v0.0.101 gateway binary.

The latest main commit inspected before filing was bafcad84803995793a85a9b9fdd3a49ef3c51145. Its only intervening commit is #8753, which does not change gateway listener attribution. The defect was not rerun at that SHA; the affected implementation is unchanged from the reproduced SHA.

Environment

  • Host: CPU-only Brev nc-trialrun-aug5 (brev-babe6wwn7)
  • OS: Ubuntu 24.04 x86_64
  • CPU: 4 vCPUs
  • GPU: none
  • Docker: client and server 29.7.1
  • Node.js: v22.23.2
  • Reproduced NemoClaw commit: cae757edca1e9996dba57d1d6b85f2d0ab1b23bb
  • Latest inspected NemoClaw commit: bafcad84803995793a85a9b9fdd3a49ef3c51145
  • OpenShell: v0.0.101
  • OpenShell CLI SHA-256: 1ad48efd5e1de8f3f017a81b3a7177872f350343a1a8d8074c7e844bca4801e9
  • cloudflared: 2026.6.1
  • E2E shard: OpenClaw MCP bridge
  • Runtime preflight: full lifecycle
  • Run time: 75.689 seconds

Debug output

The gateway evidence showed:

  • HTTP health response: 200
  • listeners: 127.0.0.1:8080 and 172.18.0.1:8080
  • process identity: openshell-gateway[nemoclaw=nemoclaw;port=8080]
  • /proc/<pid>/exe: expected v0.0.101 gateway binary
  • trusted-binary gate: rejected
  • target-bound gateway matcher: accepted

The run stopped before MCP bridge configuration or tool discovery. It therefore does not reproduce #8746 and does not establish an OpenShell MCP defect.

The run stored ephemeral MCP TLS private keys at /home/ubuntu/p014-main-cae757ed/tmp/nemoclaw-mcp-tls/{ca.key,server.key} with mode 0600. The test process used them for this one run.

NemoClaw stored the gateway JWT signing key at /home/ubuntu/p014-main-cae757ed/home/.local/state/nemoclaw/openshell-docker-gateway/jwt/signing.pem and the gateway TLS CA private key at /home/ubuntu/p014-main-cae757ed/home/.local/state/nemoclaw/openshell-docker-gateway/tls/ca.key. Both had mode 0600. The gateway used them for this one run. Only the ubuntu account and privileged host processes could access these private keys and the MCP TLS keys.

MCP fixture credentials never entered an OpenShell provider because onboarding stopped first. Cleanup stopped the verified gateway process and removed the isolated home, gateway state, all named private keys, runner temporary directory, and isolated cloudflared binary. No test container or gateway listener remained, and the before/after UFW records were identical.

Acceptance criteria

  • Treat openshell-gateway[nemoclaw=nemoclaw;port=8080] as the expected target only when the kernel-backed executable path matches the trusted gateway binary, the existing Docker-driver environment predicate passes, and the Linux process start-time and /proc/<pid>/exe samples remain stable before and after classification.
  • Preserve fail-closed behavior for an executable mismatch, missing required environment evidence, a different gateway target, and an unowned port listener.
  • Add focused regression coverage for the exact tagged identity observed on OpenShell v0.0.101.
  • Run the complete OpenClaw MCP live E2E on current main with OpenShell v0.0.101.
  • Pass trusted-private discovery, later DNS-rebinding denial, WebSocket coverage, remaining network assertions, redacted diagnostic persistence, secret scans, and cleanup.

Checklist

  • I confirmed this bug is reproducible.
  • I searched existing issues and this is not a duplicate.

Duplicate boundary: #8746 reached MCP tool discovery and returned zero tools. This issue stops in NemoClaw onboarding before MCP configuration. Searches of open and closed issues found no match for the exact readiness error or tagged gateway identity.

Activity

  1. senthilr-nv
    added
    needs: triageAwaiting maintainer classification
    on Aug 10, 2026
  2. senthilr-nv
    added the
    Bug
    issue type on Aug 10, 2026
  3. senthilr-nv
    self-assigned this
    on Aug 10, 2026
  4. jyaunches
    added a commit that references this issue on Aug 10, 2026

    fix(cli): accept tagged trusted gateway identity (#8757)

    63f8847
  5. jyaunches
    closed this as completedin #8757on Aug 10, 2026

  6. wangericnv commented on Aug 12, 2026

    @wangericnv
    wangericnv
    on Aug 12, 2026
    Collaborator

    @senthilr-nv The same two capabilities still resolve to unknown on v0.0.107, which was tagged about 27 hours after this issue was closed — so the fix should be in the build I tested. Reporting here rather than opening a new issue because the symptom string is identical, but the shape of my repro differs from yours in one way that may matter.

    Difference from the original report: yours had a gateway that NemoClaw had just started, returning HTTP 200 and owning the listeners, and the trusted-binary gate rejected its tagged identity. In mine there is no gateway at all — no registration, no process, nothing listening on the port — and readiness still cannot decide.

    $ openshell gateway list
No gateways found.

$ ss -tlnp | grep ':8080'
(no output)

$ docker ps -a --format '{{.Names}}'
(no output)

    Probe result on that host

    status = inconclusive, exitCode = 3

capability gateway.authority.resolved  = present
capability gateway.attachment.valid    = present
capability gateway.reuse.ready         = present
capability gateway.version.compatible  = unknown
capability gateway.port.uncontested    = unknown

observation gateway.owner.port    = 8080
observation gateway.reuse         = 'missing'
observation gateway.version_drift = None
observation gateway.port_conflict = None

evidence: {"id": "gateway.port.conflict",
           "summary": "Gateway port 8080 could not be observed completely.
                       Inspect port 8080 and stop only its owning process before retrying."}

    The guidance is unactionable in this state — there is no owning process to stop.

    Onboarding is blocked by it

    $ NEMOCLAW_PROVIDER=ollama NEMOCLAW_OLLAMA_MODEL=qwen3.5:9b nemoclaw onboard --fresh \
    --name wsl-probe --non-interactive --yes --yes-i-accept-third-party-software
...
✗ System readiness could not confirm required capabilities: gateway.version.compatible, gateway.port.uncontested.
Onboarding did not finish.
exit=1

    The fix does appear to be present in the build

    $ grep -rho 'openshell-gateway\[[^]]*\]' ~/.nemoclaw/source/dist
openshell-gateway[nemoclaw=${gatewayName};port=${port}]

    lib/readiness/gateway-production.js has ownedHostGatewayTarget() and the gatewayProcessSamplesMatchTrustedBinary() path, so this does not look like a build that predates the change.

    Platform contrast

    Three hosts, all NemoClaw v0.0.107 (v0.0.107-0-g03ccdd74), all with zero gateways at probe time:

    Host host.os.wsl gateway.version.compatible gateway.port.uncontested nemoclaw onboard
    WSL2, Docker Desktop integration, no systemd true unknown unknown refused
    Ubuntu x86_64, native Docker 29.5.2 false present present succeeds
    DGX GB300, Ubuntu aarch64, Docker 29.2.1 false present present succeeds

    Environment of the failing host: Ubuntu 24.04.4 (WSL2), x86_64, Docker 29.5.3 via Docker Desktop integration (host.platform.wsl_docker_desktop = present, host.platform.wsl_native_docker = absent), OpenShell 0.0.101, no systemd (systemctl --user is-system-running → offline). The same host onboarded normally on v0.0.103, which has no readiness admission gate.

    Happy to open a separate issue if you consider the no-gateway WSL path distinct from what was fixed here — I did not want to file a duplicate given the identical failure text.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

needs: triageAwaiting maintainer classification

Type

Bug

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions