Skip to content

fix(server): preserve management baselines during startup backfill - #3797

Open
shiju-nv wants to merge 1 commit into
NVIDIA:mainfrom
shiju-nv:fix/startup-conflict-recovery
Open

shiju-nv wants to merge 1 commit into
NVIDIA:mainfrom
shiju-nv:fix/startup-conflict-recovery

Conversation

@shiju-nv

@shiju-nv shiju-nv commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

A startup image-policy upload can read an empty sandbox baseline, then wait behind an operator's repair. When that upload commits, it can replace the repair. This preserves the management baseline and makes startup read fresh configuration after a write conflict.

This covers a backfill selected before the competing repair. General fencing of delayed startup snapshots remains separate work.

Related Issue

Related to #3784 and #3785, which addressed refused writes and stale rejection reports. This accepted-backfill race needs its own tracking issue before landing.

Changes

  • Restrict replacement of an existing first-start baseline to management callers, including the matching-revision path.
  • Refetch after ABORTED during image-policy upload and filesystem-baseline write-back, preserving transport retries and the reconciliation budget.
  • Add one real gateway regression that queues a repair before an image upload behind the same mutation lock. It covers Pending and Rejected admission, preserved history, rollback and a fresh successful sync.
  • Clarify the bounded ownership guarantee in the sandbox documentation.

Release Notes

An operator's policy repair now survives a startup backfill that was already waiting to write.

Testing

  • Checks appropriate to the affected code and behavior pass
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable)

The new gateway regression passes on the candidate and fails on main at c92c2c798: main accepts the stale upload as revision 2. Scoped Clippy passes for the server and supervisor, including their test targets. Formatting, SPDX and documentation checks pass. E2E coverage is requested with test:e2e; no final-head E2E pass is claimed yet.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

@copy-pr-bot

copy-pr-bot Bot commented Sep 28, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@shiju-nv
shiju-nv force-pushed the fix/startup-conflict-recovery branch from b4a4566 to b2675ba Compare October 4, 2026 15:29
@shiju-nv
shiju-nv force-pushed the fix/startup-conflict-recovery branch from b7664dc to 20879b9 Compare October 13, 2026 14:06
@shiju-nv shiju-nv changed the title fix(supervisor): refetch startup configuration after conflicts fix(server): keep an operator's startup policy repair over a stale supervisor write Oct 13, 2026
@shiju-nv
shiju-nv marked this pull request as ready for review October 13, 2026 14:08
@github-actions

Copy link
Copy Markdown

This pull request has had no activity for 14 days and is now marked stale.

@github-actions github-actions Bot added the state:stale Inactive item at risk of automatic closure. label Oct 28, 2026
@shiju-nv shiju-nv added the test:e2e Requires end-to-end coverage label Nov 2, 2026
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown

Label test:e2e applied for 3664c2e. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@shiju-nv
shiju-nv force-pushed the fix/startup-conflict-recovery branch from 3664c2e to 7c820f3 Compare November 2, 2026 14:46
@shiju-nv shiju-nv changed the title fix(server): keep an operator's startup policy repair over a stale supervisor write fix(server): preserve management baselines during startup backfill Nov 2, 2026
@github-actions github-actions Bot removed the state:stale Inactive item at risk of automatic closure. label Nov 3, 2026
A startup image-policy upload can read an empty sandbox baseline, then wait behind an operator's repair. When that upload commits, it can replace the repair. This preserves the management baseline and makes startup read fresh configuration after a write conflict.

This covers a backfill selected before the competing repair. General fencing of delayed startup snapshots remains separate work.

Signed-off-by: Shiju <shiju@nvidia.com>
@shiju-nv
shiju-nv force-pushed the fix/startup-conflict-recovery branch from 7c820f3 to e723e90 Compare November 3, 2026 11:46

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants