Skip to content

fix(supervisor): report isolation backend Attach refusals to the gateway - #4791

Open
shiju-nv wants to merge 1 commit into
NVIDIA:mainfrom
shiju-nv:fix/4739-attach-refusal-report/shiju-nv
Open

shiju-nv wants to merge 1 commit into
NVIDIA:mainfrom
shiju-nv:fix/4739-attach-refusal-report/shiju-nv

Conversation

@shiju-nv

Copy link
Copy Markdown
Collaborator

Summary

When an isolation backend refused Attach, the supervisor exited without telling the gateway. The configuration stayed ConfigurationReady: True (ConfigurationAccepted), and a revision the supervisor had written at startup stayed Pending with no error. The supervisor now reports the refusal: the gateway records the accepted configuration as rejected, and a revision this start wrote ends Failed with the backend's message.

Related Issue

Closes #4739

Changes

  • proto/openshell.proto: ReportSandboxConfigurationRequest gains bool refused_before_start = 4. It is set only with a Rejected admission for the generation that instance already reported Accepted. Omitted or false keeps today's behavior.
  • crates/openshell-server/src/grpc/policy.rs: handle_report_sandbox_configuration moves an Accepted configuration to Rejected only when the flag is set and the report names the same instance and generation. The message is gateway-authored; runtime text stays out of public status, as before. The flag with any other state is INVALID_ARGUMENT. Every unflagged report behaves as before, including a runtime that fails to apply the generation it is running (for example unreachable middleware), which keeps the accepted state.
  • crates/openshell-core/src/grpc_client.rs: report_sandbox_configuration delegates to the new report_sandbox_configuration_request, which sends a caller-built request. The span name is unchanged.
  • crates/openshell-supervisor/src/backend_setup.rs: attach tags Denied and Attach errors as refusals. Descriptor (driver and TLS faults) and Unavailable are not tagged.
  • crates/openshell-supervisor/src/lib.rs: on a tagged Attach error, run_sandbox_with_backend sends the flagged Rejected report for the accepted generation (one bounded attempt), then reports a revision this start wrote (image discovery or baseline-path write-back) Failed with the backend's message. A revision the user wrote is not marked. The original error is still returned.
  • crates/openshell-server/src/storage_proto.rs: PUBLIC_RPC_SCHEMA_SHA256 for the new public-only field, taken from the failing test output. Durable schemas are unchanged.
  • sdk/go/proto/openshellv1/openshell.pb.go: regenerated with mise run go:proto:gen.
  • docs/extensibility/isolation-backends.mdx: what users see after a refusal.

Testing

  • cargo test -p openshell-supervisor --lib (177 passed), cargo test -p openshell-server --lib (2338 passed, 21 ignored, including public_and_durable_schema_inventories_are_complete), cargo test -p openshell-core --lib (574 passed)
  • cargo clippy -p openshell-supervisor -p openshell-server -p openshell-core --all-targets -- -D warnings, cargo fmt --check, go build ./... in sdk/go, mise run go:proto:check
  • Unit tests added:
    • attach_refusal_is_marked_as_a_refusal: a refused Attach carries the refusal code.
    • attach_refusal_reports_the_accepted_generation_as_refused: the report names the accepted instance and generation and sets refused_before_start; transient errors, local policy and a missing instance send nothing.
    • attach_refusal_fails_only_the_revision_this_start_wrote.
    • configuration_admission_records_refusal_of_accepted_generation: an unflagged same-generation rejection keeps Accepted, the flag with Accepted is INVALID_ARGUMENT, and a flagged rejection becomes ConfigurationInvalid without runtime text.
    • Each fails with its fix removed (checked by mutation).
  • E2E tests added/updated (if applicable): none. Checked by hand with examples/isolation-backend-v8, whose backend refuses Attach when process.run_as_user differs from the host identity:
    • with the policy run_as_user: "1500" on a uid 502 host, openshell sandbox get shows ConfigurationReady: False (ConfigurationInvalid) with the gateway message;
    • on main the same run shows ConfigurationReady: True (ConfigurationAccepted).

Recovery

Delete the sandbox and create it again with a policy the backend accepts. A refused sandbox stays in Provisioning until its repair deadline, and sandbox start, sandbox stop and a process-policy change are all refused in that phase, as on main.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. rust-sdk — No explicit risk acceptance found

ReportSandboxConfigurationRequest gains refused_before_start, so its publicly re-exported prost Rust struct now has a fourth required struct-literal field instead of three.

  • Impact: Rust consumers constructing openshell_sdk::raw::proto::ReportSandboxConfigurationRequest { ... } without ..Default::default() fail to compile because refused_before_start is missing.
  • Action: Remove the field from v0.1.6 and introduce it in a minor release with migration guidance for Rust raw callers to add refused_before_start: false or ..Default::default().
  • PR: fix(supervisor): report isolation backend Attach refusals to the gateway #4791
Review details
  • Execution: complete
  • Candidate: PR #4791 (ad0916e8d87c8ebd32c53bac3dca78208b750ca8)
  • Baseline: d22394370d2d8b27859ffa772cac3aaaa79feb2c (d22394370d2d8b27859ffa772cac3aaaa79feb2c)
  • Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

@shiju-nv
shiju-nv force-pushed the fix/4739-attach-refusal-report/shiju-nv branch from 085e666 to 882780f Compare October 21, 2026 07:28
@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

2. typescript-sdk — No explicit risk acceptance found

@nvidia/openshell-sdk/raw's generated ReportSandboxConfigurationRequest previously lacked refusedBeforeStart but now requires that boolean property on complete message values.

  • Impact: Existing complete object literals typed as ReportSandboxConfigurationRequest fail TypeScript checking for missing refusedBeforeStart, while create() and partial RPC initializers remain compatible.
  • Action: Defer this required-property addition from v0.1.7 to a minor release with notice to add refusedBeforeStart: false or use create(ReportSandboxConfigurationRequestSchema, init).
  • PR: fix(supervisor): report isolation backend Attach refusals to the gateway #4791
Review details
  • Execution: complete
  • Candidate: PR #4791 (cda1f4fd5879a70b028ae3bbc11261568cbebde0)
  • Baseline: 42671c2dd2c02b744850e0bda72b99d5509969c1 (42671c2dd2c02b744850e0bda72b99d5509969c1)
  • Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. rust-sdk — No explicit risk acceptance found

Public raw protobuf structs ReportSandboxConfigurationRequest and UpdateConfigResponse now require refused_before_start and created_policy_revision, respectively, in previously complete Rust literals.

  • Impact: Existing request constructors and mock/server response constructors fail to compile unless they initialize the new fields or use struct-update defaults.
  • Action: Preserve both existing raw message shapes for v0.1.7 and defer the documented false-field or Default initialization migration to a qualified minor train.
  • PR: fix(supervisor): report isolation backend Attach refusals to the gateway #4791

2. typescript-sdk — No explicit risk acceptance found

Generated raw ReportSandboxConfigurationRequest and UpdateConfigResponse types now require refusedBeforeStart and createdPolicyRevision, respectively, where neither property previously existed.

  • Impact: Existing complete request literals and mock response values fail TypeScript checking, while generated create() builders and partial RPC initializers remain compatible.
  • Action: Preserve both existing raw message shapes for v0.1.7 and defer the documented boolean-default or create() migration to a qualified minor train.
  • PR: fix(supervisor): report isolation backend Attach refusals to the gateway #4791
Review details
  • Execution: complete
  • Candidate: PR #4791 (0c86101ef8de2a2f4fd6716b347550adaf6fa3b2)
  • Baseline: 42671c2dd2c02b744850e0bda72b99d5509969c1 (42671c2dd2c02b744850e0bda72b99d5509969c1)
  • Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

@shiju-nv shiju-nv added the test:e2e Requires end-to-end coverage label Nov 2, 2026
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown

Label test:e2e applied for dc58469. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@shiju-nv
shiju-nv force-pushed the fix/4739-attach-refusal-report/shiju-nv branch from dc58469 to 5c45561 Compare November 2, 2026 14:46
copy-pr-bot Bot pushed a commit that referenced this pull request Nov 2, 2026
Classify Build, Attach and Confirm failures with fixed public diagnostics and a bounded reporting budget while preserving the original startup error.

Closes #5107

Depends on corrected PR #4791.

Signed-off-by: Shiju <shiju@nvidia.com>
When an isolation backend refused Attach, the supervisor exited without telling the gateway. The configuration stayed `ConfigurationReady: True (ConfigurationAccepted)`, and a revision the supervisor had written at startup stayed Pending with no error. The supervisor now reports the refusal: the gateway records the accepted configuration as rejected, and a revision this start wrote ends Failed with the backend's message.

Signed-off-by: Shiju <shiju@nvidia.com>
@shiju-nv
shiju-nv force-pushed the fix/4739-attach-refusal-report/shiju-nv branch from 5c45561 to 37d1373 Compare November 3, 2026 11:46

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Show the backend reason when it rejects a sandbox policy during Attach

1 participant