Skip to content

fix(supervisor): report bounded backend startup failures - #5118

Open
shiju-nv wants to merge 2 commits into
NVIDIA:fix/4739-attach-refusal-report/shiju-nvfrom
shiju-nv:fix/5107-bounded-backend-startup-status/shiju-nv
Open

shiju-nv wants to merge 2 commits into
NVIDIA:fix/4739-attach-refusal-report/shiju-nvfrom
shiju-nv:fix/5107-bounded-backend-startup-status/shiju-nv

Conversation

@shiju-nv

@shiju-nv shiju-nv commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

A backend can fail during construction, Attach or Confirm before the workload starts. This gives operators a fixed failure stage and recovery hint in sandbox status, keeps detailed errors in supervisor logs and bounds the time spent reporting the failure.

Related Issue

Closes #5107. Part of #5114.

Depends on corrected PR #4791. Review this layer against fix/4739-attach-refusal-report/shiju-nv. For the fork-based series, land the parent first, then replay only this layer onto the resulting main.

Changes

  • Send only recognized diagnostic codes through authenticated startup reports checked against the admitted generation.
  • Bound the complete best-effort report operation and preserve the original startup error if reporting fails.
  • Treat only a typed Attach denial as policy refusal. Build or Confirm denials and generic Attach failures do not fail startup-owned policy history.
  • Preserve takeover confirmation, network ordering and the parent's corrected policy-write receipt.

Release Notes

Backend startup failures expose a bounded recovery hint before workload execution. Follow the hint and wait for cleanup before retrying or recreating the sandbox. This layer inherits PR #4791's next-minor-only API requirement.

Testing

  • Checks appropriate to the affected code and behavior pass
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. rust-sdk — No explicit risk acceptance found

The SDK-exposed ReportSandboxConfigurationRequest and UpdateConfigResponse structs gain required refused_before_start and created_policy_revision fields, respectively.

  • Impact: Existing complete literals in raw clients and mock/server implementations fail with missing-field errors, although RPC signatures and legacy wire payloads remain compatible.
  • Action: Preserve the baseline raw struct shapes in v0.1.7 and defer these additions to the documented minor-release migration.
  • PR: fix(supervisor): report bounded backend startup failures #5118

2. typescript-sdk — No explicit risk acceptance found

Generated raw ReportSandboxConfigurationRequest and UpdateConfigResponse types now require refusedBeforeStart and createdPolicyRevision properties that baseline complete literals did not need.

  • Impact: Consumers constructing complete raw messages or typed mock responses fail TypeScript checking, while generated builders and partial-message RPC callers remain compatible.
  • Action: Keep complete raw message literals source-compatible in v0.1.7 and defer the required properties to the documented minor-release migration.
  • PR: fix(supervisor): report bounded backend startup failures #5118
Review details
  • Execution: complete
  • Candidate: PR #5118 (04c33f7a881e629f69fb4a6aef38a5741fc729f3)
  • Baseline: c92c2c7983b1307715f6a922366dd3759e41b35b (c92c2c7983b1307715f6a922366dd3759e41b35b)
  • Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

@shiju-nv shiju-nv added the test:e2e Requires end-to-end coverage label Nov 2, 2026
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown

Label test:e2e applied for 4c8fa9d. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@shiju-nv
shiju-nv force-pushed the fix/5107-bounded-backend-startup-status/shiju-nv branch from 4c8fa9d to 449ef90 Compare November 2, 2026 14:46
@shiju-nv
shiju-nv changed the base branch from main to fix/4739-attach-refusal-report/shiju-nv November 2, 2026 14:46
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. api-behavior — No explicit risk acceptance found

Backend transport and availability failures previously remained infrastructure failures; they now report rejected admission and ConfigurationReady=False with reason ConfigurationInvalid.

  • Impact: Controllers and SDK consumers using ConfigurationInvalid to select configuration repair instead of infrastructure recovery now misclassify valid configurations during backend outages.
  • Action: Preserve infrastructure-failure classification in v0.1.7 and defer the reclassification to a minor release with migration guidance for readiness-based recovery automation.
  • PR: fix(supervisor): report bounded backend startup failures #5118
Review details
  • Execution: complete
  • Candidate: PR #5118 (bc39e072e588fde35c89d6f5052d167cddfd6d5b)
  • Baseline: 5c4556104ccbaf0e8c2830792c79c4763e1f9c32 (5c4556104ccbaf0e8c2830792c79c4763e1f9c32)
  • Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

A backend can fail during construction, Attach or Confirm before the workload starts. This gives operators a fixed failure stage and recovery hint in sandbox status, keeps detailed errors in supervisor logs and bounds the time spent reporting the failure.

Signed-off-by: Shiju <shiju@nvidia.com>
@shiju-nv
shiju-nv force-pushed the fix/4739-attach-refusal-report/shiju-nv branch from 5c45561 to 37d1373 Compare November 3, 2026 11:46
@shiju-nv
shiju-nv force-pushed the fix/5107-bounded-backend-startup-status/shiju-nv branch from 449ef90 to cf39c71 Compare November 3, 2026 11:46
Signed-off-by: Shiju <shiju@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Report which backend startup stage failed without exposing private logs

1 participant