Skip to content

fix(vm): boot a digest-pinned image from the cache without a registry call - #4582

Merged
pimlock merged 1 commit into
NVIDIA:mainfrom
balajinvda:fix/4581-vm-digest-pinned-cache-hit/balajinvda
Oct 22, 2026
Merged

pimlock merged 1 commit into
NVIDIA:mainfrom
balajinvda:fix/4581-vm-digest-pinned-cache-hit/balajinvda

Conversation

@balajinvda

@balajinvda balajinvda commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A VM sandbox boot from the registry authenticated with the registry before it looked at the root disk cache, on every boot, even for a digest-pinned image whose root disk is cached. This returns the cache hit first for digest-pinned references, so such a boot makes no registry request and keeps working while the registry is unreachable.

Related Issue

Closes #4581

Changes

  • ensure_cached_registry_rootfs_image: when the reference carries a digest and image_cache_rootfs_image(bootstrap_image_cache_identity(digest)) exists, return the cache hit before client.auth. Authentication and the pull stay on the miss path; tags still resolve through the registry.
  • The cache-hit events (Pulling, CacheHit, Pulled) move into publish_registry_cache_hit, shared by both hit paths, so a digest-pinned hit reports exactly as before.
  • Safety: a digest names the content, and since fix(vm): bind cached registry images to verified manifest digests #4342 a cached root disk is published only after its manifest was verified against that digest, so the registry cannot add anything on a hit. Behaviour change: a host whose registry credentials were revoked still boots digest-pinned images it has already cached, as a kubelet does with IfNotPresent and a digest-pinned image.
  • Measured with openshell-driver-vm v0.1.3 (e1f3c82c) on arm64 hosts against a remote registry with the image cached: registry authentication and resolution were about 1.9 s (p50, 96 boots) of a 2.5 s boot (numbers in VM driver authenticates with the registry on every boot, even for a cached digest-pinned image #4581). The same code path authenticates on every boot in v0.1.4 (5a7728c3) and on main at 909f336b6, which this change is based on; since fix(vm): bind cached registry images to verified manifest digests #4342 a digest pin skips only the manifest request there, not the authentication.

Testing

  • Checks appropriate to the affected code and behavior pass: cargo fmt --check, cargo clippy -p openshell-driver-vm --all-targets -- -D warnings, cargo test -p openshell-driver-vm
  • Unit tests added/updated (if applicable): bootstrap_registry_cache_hit_for_a_digest_pin_makes_no_registry_request (stub registry): a miss authenticates and fetches; a cached digest pin makes no request; a tag for the same content still resolves. Without the change it fails with ["GET /v2/"].
  • E2E tests added/updated (if applicable)

cargo test -p openshell-driver-vm also fails stopping_a_sandbox_lets_the_supervisor_drain_before_closing_liveness on the test machine; it fails the same way on unpatched main, so it is not related to this change.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

… call

A registry-sourced sandbox boot authenticated with the registry, and for a
tag resolved the manifest, before it looked at the root disk cache. It did
so on every boot, even when the image reference carries a digest and the
root disk for that digest is already cached. Each boot then waits on the
registry, and a registry outage stops every boot of an image the host
already has.

A digest names the image's content, and since NVIDIA#4342 a cached root disk is
published only after its manifest was verified against that digest. On a
cache hit for a digest-pinned reference the registry cannot add anything,
so return the hit before authenticating. Authentication and the pull stay
on the miss path, and a tag still resolves through the registry. The hit
emits the same Pulling, CacheHit and Pulled events as before.

Behaviour change: a host whose registry credentials were revoked still
boots digest-pinned images it has already cached, as a kubelet does with
IfNotPresent and a digest-pinned image.

Test: bootstrap_registry_cache_hit_for_a_digest_pin_makes_no_registry_request
covers a miss (authenticates and fetches), a cached digest pin (no request
at all) and a tag for the same content (still resolved). It fails without
this change with ["GET /v2/"].

Closes NVIDIA#4581

Signed-off-by: Balaji Ganesan <bganesan@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 15, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@pimlock

pimlock commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

/ok to test 596bc12

@pimlock pimlock self-assigned this Oct 21, 2026
@pimlock
pimlock self-requested a review October 21, 2026 21:54
@pimlock pimlock added the test:e2e Requires end-to-end coverage label Oct 21, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 596bc12. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@pimlock

pimlock commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

gator-agent

PR Review Status

The code-only review found no blocking correctness or security findings in the two-file change. The digest-pinned bootstrap fast path uses the existing verified, versioned cache identity, preserves cache-hit progress events, and has a focused regression test for cache misses, zero-request digest hits, and tag resolution.

Action required: A maintainer must accept linked issue #4581 with state:accepted, rerun PR Issue Acceptance, and select Re-run all jobs on E2E run 39674801874. Before pipeline handoff, the author should update the MicroVM section of docs/how-it-works/sandboxes/runtimes.mdx, or a maintainer should explain why docs are intentionally unnecessary.

The acceptance check fails because the author lacks repository write access and #4581 is not accepted. The E2E label is present, but the existing run completed with its test jobs skipped; the label-help bot explicitly requests a rerun. Gator is blocked on these process steps and has not rerun workflows under this invocation.

Blocking code findings: None.

Carried findings: None.

Non-blocking suggestion:

  • Document that a cached digest-pinned bootstrap disk skips registry authentication and resolution, including after credential revocation. Tags, cache misses, and distinct registry-sourced workload images retain their existing registry-access requirements. No navigation change is needed. This is also the workflow's documentation requirement before pipeline handoff.
Gator metadata
  • Validation: Focused VM-driver cache reliability and boot-latency fix implementing VM driver authenticates with the registry on every boot, even for a cached digest-pinned image #4581; no duplicate active implementation found.
  • Docs: Published MicroVM behavior update or explicit maintainer explanation required before pipeline handoff.
  • Checks: Branch Checks, Helm Lint, Trivy Changes, DCO, and vouch pass; PR Issue Acceptance fails.
  • E2E: test:e2e present; standard E2E jobs skipped in completed attempt 1, rerun required. No GPU or Windows label needed.
  • Head SHA: 596bc12fe8e45fa29d8c862f7693dec2444d5d23
  • Base SHA: 909f336b624a1b7efc5a0c607e35e20dcb96923e
  • Merge base SHA: 909f336b624a1b7efc5a0c607e35e20dcb96923e
  • Patch ID: d81199ed4e11cabbd4dce0613f77f847a313c7d1
  • Gator payload: 12
  • Review mode: initial
  • Previous reviewed SHA: none
  • Review budget exhausted: no
  • Maintainer decision required: no convergence decision
  • Next state: gator:blocked
  • Blocked reason: test_dispatch_required

@pimlock pimlock added the gator:blocked Gator is blocked by process or repository gates label Oct 21, 2026
@pimlock
pimlock added this pull request to the merge queue Oct 22, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 22, 2026
@pimlock
pimlock added this pull request to the merge queue Oct 22, 2026
Merged via the queue into NVIDIA:main with commit 9152b6e Oct 22, 2026
155 of 157 checks passed
@balajinvda
balajinvda deleted the fix/4581-vm-digest-pinned-cache-hit/balajinvda branch October 22, 2026 18:08
@pimlock

pimlock commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

gator-agent

Monitoring Complete

Monitoring is complete because this PR has merged.

Final status: The existing code review found no blocking findings, and maintainer approval was present. The active gator:blocked label is now stale because the PR has merged.

I removed the active gator:* label because there is nothing left for gator to monitor on this PR.

Gator metadata
  • Head SHA: 596bc12fe8e45fa29d8c862f7693dec2444d5d23
  • Gator payload: 12

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

VM driver authenticates with the registry on every boot, even for a cached digest-pinned image

2 participants