[sig-provider] lease-shell exec against a closed/terminated lease returns synthetic result{exit_code:0} (no output, no failure frame)

[jobordu]

Summary

Over a mainnet lease-shell exec, running a command against a lease whose on-chain state is closed/terminated returns a synthetic success: a single result frame decoding to {"exit_code":0} with no stdout, no stderr, and no failure frame — for every command, deterministically. A client that trusts rc == 0 therefore reads a dead lease as a healthy one that "ran the command and produced no output."

This is distinct from (and more clear-cut than) the transient trailing-stdout loss discussed in akash-network/console#3438; it surfaces through the same Console provider-proxy exec path, which is why that issue is being kept open there.

Environment

• Mainnet, WebSocket lease-shell exec, non-interactive (tty=false), via the Console provider-proxy.
• Reproduced on a mainnet provider (anonymized "suspect"), with a healthy mainnet provider ("control") as the positive control, exercised through the identical client code path.

Frame protocol

Each relayed message is tagged by a leading byte: 100=stdout, 101=stderr, 102=result (JSON exit code), 103=failure, 104=stdin, 105=resize.

Evidence

Same client, same commands, two providers:

Control (healthy lease — lease.state=active, service available=1, logs show the container's startup line):

command	rc	stdout	frames
echo HELLO_WORLD	0	HELLO_WORLD\n	[100 "HELLO_WORLD\n", 102 {"exit_code":0}]
whoami	0	root\n	[100 "root\n", 102 {"exit_code":0}]

Suspect (lease terminated — lease.state=closed, no service reported, container logs empty):

command	rc	stdout	stderr	frames
echo, whoami, cat /etc/os-release, seq 1 200, 64 KB blob, hostname; id, …	0	(empty)	(empty)	[102 {"exit_code":0}] only

• The 102 payload is literally {"exit_code":0}\n (16 bytes) for every command. No 100, no 101, and crucially no 103 failure ever.
• Outbound capture confirms the command bytes sent to the suspect are byte-identical to the control (e.g. cmd0=whoami), so this is not a client-side transmission problem.
• Container liveness was checked out-of-band via three independent paths that do not touch the shell endpoint — lease status (available), the logs endpoint, and the events endpoint — plus the raw lease.state. All three corroborate: the control is alive (available=1, logs present); the suspect lease is closed with nothing serving.

Repro caveat: the exec runs an argv vector (cmd0,cmd1,…), not a shell string, so exit 7 is exec-not-found (there is no /bin/exit) and is not a valid exit-code-propagation test — it returns rc=0 on a healthy lease too. Use real binaries (whoami, cat) when validating output.

Expected vs actual

• Expected: exec against a closed/terminated lease should fail — a 103 failure frame, a connection error, or a non-zero result — so a caller can tell the lease is not serving.
• Actual: a synthetic {"exit_code":0} with no output and no failure frame, indistinguishable at the rc level from a command that ran and printed nothing.

Impact

rc == 0 is not a reliable health signal for lease-shell exec: a terminated lease reports success. Any client that runs a command and checks the exit code (health checks, one-shot commands, CI probes, the CLI/SDK) can read a dead lease as healthy. The practical mitigation we use downstream is to require a known marker echoed back in stdout rather than trusting rc==0.

Notes

• Providers anonymized (control/suspect) intentionally.
• I did not determine why the suspect lease closed (the container never emitted its startup line, so it appears to have never run the workload); this report is scoped to the exec handler's behavior against an already-closed lease, which is directly reproducible.

[jobordu]

Closing this for now pending fuller characterization. I reproduced the closed-lease synthetic-success (rc=0, no output, no failure frame) deterministically, but only on a single provider, and I have not yet established why the lease closed — I'd rather confirm it across more providers and nail the precondition before asserting a provider defect. Will reopen with the complete data set. Apologies for the churn.