Skip to content

fix: move container registry to nvcr#651

Open
mckornfield wants to merge 2 commits into
mainfrom
nvcr-reg-not-ghcr/mck
Open

fix: move container registry to nvcr#651
mckornfield wants to merge 2 commits into
mainfrom
nvcr-reg-not-ghcr/mck

Conversation

@mckornfield

@mckornfield mckornfield commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Set up arm variant
  • Push to nvcr instead of ghcr

Pre-Review Checklist

Ensure that the following pass:

  • mise run format && mise run check or via prek validation.
  • mise run test passes locally
  • mise run test:e2e passes locally
  • mise run test:ci-container passes locally (recommended)
  • GPU CI status check passes -- comment /sync on this PR to trigger a run (auto-triggers on ready-for-review)

Pre-Merge Checklist

  • New or updated tests for any fix or new behavior
  • Updated documentation for new features and behaviors, including docstrings for API docs.

Other Notes

  • Closes #

Summary by CodeRabbit

  • New Features

    • GPU container images are now built for both Linux AMD64 and ARM64 architectures.
    • Release-tagged images are published to the NVIDIA container registry.
  • Documentation

    • Updated workflow diagrams and container documentation with the new registry location, image naming, release behavior, and multi-platform usage instructions.

Signed-off-by: Matt Kornfield <mkornfield@nvidia.com>
@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

The container workflow now builds linux/amd64 and linux/arm64 GPU images, publishes release images to NVCR, and uses GHCR for BuildKit caching. Workflow diagrams and container documentation reflect the new registry and multi-platform manifest behavior.

Changes

Container publishing

Layer / File(s) Summary
Workflow registry and multi-architecture build
.github/workflows/container-build.yml, .github/workflows/README.md
The workflow builds both target architectures, authenticates with NVCR, uses GHCR for caching, and updates the workflow overview to show NVCR publication.
Documentation and multi-architecture examples
docs/developer-guide/docker.md, containers/README.md
Image repositories, release-tag manifest descriptions, and multi-architecture build examples now use the NVCR repository.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested labels: chore

Suggested reviewers: zywind

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: moving container registry usage to NVCR.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch nvcr-reg-not-ghcr/mck

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot added the chore Maintenance not tied to a user-visible change label Jul 15, 2026
@greptile-apps

greptile-apps Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR migrates the published GPU container registry from GHCR to nvcr.io/aire/safe-synth-dev and adds Linux ARM64 support via QEMU emulation. GHCR is retained exclusively for BuildKit cache layers. A dual-secret fallback (NVCR_AIRE_SAFE_SYNTHESIZER_DEPLOY_KEY || NVCR_AIRE_SAFE_SYN_DEPLOY_KEY) is introduced to smooth the repository-secret rotation.

  • Registry migration: REGISTRY_IMAGE is now nvcr.io/aire/safe-synth-dev/safe-synthesizer; CACHE_IMAGE is introduced to keep the GHCR cache reference separate. All docs, README diagrams, and developer-guide examples are updated to match.
  • ARM64 support: QEMU is set up unconditionally, and the build matrix now targets linux/amd64,linux/arm64 for the cu129 variant on both push and workflow_dispatch events.
  • Secret guard: A new "Check NVCR deploy key" step fails fast if neither secret resolves to a non-empty value before the NVCR login step runs.

Confidence Score: 5/5

Safe to merge — the registry migration is well-structured with a fail-fast secret check, and all documentation is consistent with the workflow changes.

The change is a straightforward registry migration: NVCR replaces GHCR as the publish target, GHCR is retained for cache only, and the split is clearly reflected in separate env vars. The fail-fast key-check step prevents silent failures if neither secret is present. Documentation across all four files is consistent and accurate.

No files require special attention beyond the previously noted manual-dispatch build-time impact in .github/workflows/container-build.yml.

Important Files Changed

Filename Overview
.github/workflows/container-build.yml Migrates publish target to NVCR, splits cache reference into CACHE_IMAGE, adds QEMU+arm64 support, dual-secret fallback, and a fail-fast key-check step. Manual dispatch builds are now slower (two platforms, no cache) — noted in prior review.
docs/developer-guide/docker.md Documentation updated to reflect NVCR registry path, multi-platform manifest details, and dual-secret rotation note; all examples are consistent with the workflow changes.
containers/README.md Registry example command updated from GHCR to NVCR; caveat about per-architecture wheel availability removed in favour of a note about the release workflow using the same multi-arch build.
.github/workflows/README.md Table and Mermaid flowchart updated from GHCR to NVCR and from single-arch to multi-arch; purely documentation.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant GH as GitHub Event
    participant WF as container-build.yml
    participant QEMU as QEMU (arm64)
    participant GHCR as GHCR (cache only)
    participant NVCR as nvcr.io/aire/safe-synth-dev

    GH->>WF: "push v* tag OR workflow_dispatch"
    WF->>QEMU: setup-qemu-action (arm64)
    WF->>WF: setup-buildx-action

    alt push event (release tag)
        WF->>WF: Check NVCR deploy key (fail-fast if missing)
        WF->>NVCR: docker/login-action ($oauthtoken)
        WF->>GHCR: docker/login-action (GITHUB_TOKEN)
        WF->>GHCR: Pull buildcache-cu129 (cache-from)
        WF->>NVCR: Build and push linux/amd64 + linux/arm64
        WF->>GHCR: "Write buildcache-cu129 (cache-to, mode=max)"
    else workflow_dispatch (validation)
        WF->>WF: Build linux/amd64 + linux/arm64 (no push, no cache)
    end
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant GH as GitHub Event
    participant WF as container-build.yml
    participant QEMU as QEMU (arm64)
    participant GHCR as GHCR (cache only)
    participant NVCR as nvcr.io/aire/safe-synth-dev

    GH->>WF: "push v* tag OR workflow_dispatch"
    WF->>QEMU: setup-qemu-action (arm64)
    WF->>WF: setup-buildx-action

    alt push event (release tag)
        WF->>WF: Check NVCR deploy key (fail-fast if missing)
        WF->>NVCR: docker/login-action ($oauthtoken)
        WF->>GHCR: docker/login-action (GITHUB_TOKEN)
        WF->>GHCR: Pull buildcache-cu129 (cache-from)
        WF->>NVCR: Build and push linux/amd64 + linux/arm64
        WF->>GHCR: "Write buildcache-cu129 (cache-to, mode=max)"
    else workflow_dispatch (validation)
        WF->>WF: Build linux/amd64 + linux/arm64 (no push, no cache)
    end
Loading

Reviews (2): Last reviewed commit: "chore: address CR" | Re-trigger Greptile

@codecov

codecov Bot commented Jul 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7770d127-e513-49f0-a3fc-0420b4f58d11

📥 Commits

Reviewing files that changed from the base of the PR and between e974bdf and 547006d.

📒 Files selected for processing (4)
  • .github/workflows/README.md
  • .github/workflows/container-build.yml
  • containers/README.md
  • docs/developer-guide/docker.md
📜 Review details
⏰ Context from checks skipped due to timeout. (9)
  • GitHub Check: Build cu129 image
  • GitHub Check: Unit Tests (3.13)
  • GitHub Check: Unit Tests (3.12)
  • GitHub Check: Unit Tests (3.11)
  • GitHub Check: End-user Wheel Install
  • GitHub Check: Typecheck
  • GitHub Check: Analyze (Python)
  • GitHub Check: Greptile Review
  • GitHub Check: Analyze (python)
🧰 Additional context used
📓 Path-based instructions (10)
**/*.{md,markdown,py}

📄 CodeRabbit inference engine (.cursor/rules/agent-markdown-style.mdc)

**/*.{md,markdown,py}: Avoid decorative bold (**text**) in list items, body text, and docstrings; use structural cues (headers, list markers, colons, backticks) for emphasis instead
Use backticks for code identifiers, paths, and CLI commands in markdown and docstrings

Files:

  • containers/README.md
  • docs/developer-guide/docker.md
**/*.{md,markdown}

📄 CodeRabbit inference engine (.cursor/rules/agent-markdown-style.mdc)

**/*.{md,markdown}: Bold is acceptable only in markdown tables where it's the conventional way to mark header-like cells in the body
Use ## headers to segment markdown sections instead of bold text
Use -- (em-dash) instead of - (hyphen) for asides in markdown

Files:

  • containers/README.md
  • docs/developer-guide/docker.md
**/*.md

📄 CodeRabbit inference engine (STYLE_GUIDE.md)

Do not use decorative bold in Markdown body text, list items, or docstrings; use single backticks for code identifiers, paths, and commands.

Use the repository's Markdown style and ensure Markdown source files have SPDX copyright headers unless excluded by .copyrightignore.

Files:

  • containers/README.md
  • docs/developer-guide/docker.md
**/*

📄 CodeRabbit inference engine (STYLE_GUIDE.md)

**/*: Every source file requires the SPDX copyright and license header appropriate to its file format.
End files with a newline, remove trailing whitespace, use one space between sentences, and keep code, comments, and docstrings within 120 characters.

**/*: All merged commits must use Conventional Commits format: <type>(<scope>): <description> or <type>: <description>, with a lowercase valid type and a description no longer than 100 characters.
All contributions require a DCO Signed-off-by trailer and a cryptographic commit signature.
Before submitting a pull request, run formatting, checks, and tests using mise run format, mise run check, and mise run test.
Branches other than main must use lowercase author-prefixed names in one of the documented forms, optionally including an issue ID and category.
Release tags must use a v prefix and PEP 440 stable or release-candidate versions, such as v1.0.0 or v0.1.0rc0; alpha versions and dashed -rc suffixes are not used.
Do not move a published release tag; create and validate a new release candidate when code changes.
Use mise run <task> for project tasks; the Makefile only bootstraps mise and provides deprecated compatibility messages.

Files:

  • containers/README.md
  • docs/developer-guide/docker.md

⚙️ CodeRabbit configuration file

**/*: Review as a senior maintainer for NeMo Safe Synthesizer. Prioritize issues that can change behavior, break user workflows, weaken privacy guarantees, hide failures, make tests unreliable, or create maintenance risk. Avoid generic style commentary unless it points to a concrete project convention that automated tools will not catch.
Comment only when the finding is actionable and tied to changed code. For each finding, state the impact, the condition that triggers it, and the smallest practical fix. Prefer one precise comment over broad advice. Do not ask for refactors outside the PR scope unless the changed code creates the problem.
Review type guidance: - Potential issue: use for correctness bugs, data loss, privacy leaks,
security risks, broken public APIs, invalid config behavior, missing
validation, hidden failures, nondeterministic tests, or CI breakage.

  • Refactor suggestion: use for local maintainability problems introduced
    by the diff when they have clear future cost, such as duplicated setup,
    unclear boundaries, over-mocking, avoidable complexity, or opaque test
    helpers.
  • Nitpick: avoid in chill mode. Do not emit formatting, import-order,
    wording, or style-only comments unless automated tools cannot catch the
    issue and it affects maintainability.

Severity guidance: - Critical: security/privacy leaks, data loss, training/test/holdout
contamination, or broken release/package/core pipeline execution.

  • Major: incorrect generation/training/evaluation behavior, broken
    CLI/SDK public API, invalid config defaults or validators, or GPU/vLLM
    cleanup and process-isolation bugs likely to fail CI or production
    runs.
  • Minor: localized bugs, missing focused tests for changed behavior, or
    bad test patterns that weaken regression coverage.
  • Trivial: small cleanup with no behavior impact. Usually suppress in
    chill mode.
  • Info: context only. Avoid unless it helps reviewers understand risk.
    Safe-Synthesizer-specific review focus: - Data ...

Files:

  • containers/README.md
  • docs/developer-guide/docker.md
**/*.{py,sh,yaml,yml,md}

📄 CodeRabbit inference engine (CONTRIBUTING.md)

All source files with .py, .sh, .yaml, .yml, or .md extensions must include SPDX copyright headers, except files listed in .copyrightignore.

Files:

  • containers/README.md
  • docs/developer-guide/docker.md
containers/**

⚙️ CodeRabbit configuration file

Review container changes against STYLE_GUIDE.md and containers/README.md. Check cache mounts, uv usage, non-root runtime behavior, CUDA settings, and copy order.

Files:

  • containers/README.md
.github/workflows/*.yml

📄 CodeRabbit inference engine (CONTRIBUTING.md)

.github/workflows/*.yml: Documentation deployment is defined in .github/workflows/docs.yml and runs when docs/, mkdocs.yml, or src/ changes are pushed to main.
GitHub Actions workflows must use the repository's atomic, read-only mise tasks for CI checks.

Files:

  • .github/workflows/container-build.yml
.github/**

⚙️ CodeRabbit configuration file

Review GitHub configuration for branch protection expectations, CODEOWNERS alignment, least privilege permissions, pinned actions where practical, and consistency with CONTRIBUTING.md.

Files:

  • .github/workflows/container-build.yml
  • .github/workflows/README.md
docs/**/*.md

📄 CodeRabbit inference engine (.cursor/rules/writing-docs.mdc)

docs/**/*.md: Use MkDocs Material admonition syntax (!!! note, !!! warning, ??? tip) for highlighting important information and collapsible sections in documentation
Use MkDocs Material tabs syntax (=== "Label") to present alternative views or language-specific examples in documentation
Use code block syntax with title and highlight line parameters (title="filename", hl_lines="2 3") for code examples in documentation
Use Mermaid diagram syntax (```mermaid flowchart, etc.) for visualizations in documentation

Classify documentation using Diátaxis and use MkDocs Material syntax for admonitions, tabs, and titled or highlighted code blocks.

Documentation pages must be Markdown files under docs/, placed in the appropriate Diátaxis subdirectory, and added to the nav: section of mkdocs.yml.

Files:

  • docs/developer-guide/docker.md
docs/**

⚙️ CodeRabbit configuration file

Review documentation as MkDocs Material content. Check Diataxis fit, accurate commands, internal links, code fences, and markdown style from STYLE_GUIDE.md.

Files:

  • docs/developer-guide/docker.md
🔇 Additional comments (7)
.github/workflows/container-build.yml (3)

33-38: LGTM!


73-95: LGTM!


136-137: LGTM!

.github/workflows/README.md (2)

16-16: LGTM!


92-93: LGTM!

Also applies to: 108-108

docs/developer-guide/docker.md (1)

190-193: LGTM! Consistent with the REGISTRY_IMAGE/multi-platform changes in container-build.yml and the CONTAINER_GPU_REGISTRY example in containers/README.md.

Also applies to: 204-204, 213-215, 286-297

containers/README.md (1)

116-121: LGTM!

- variant: cu129
extra: cu129
platforms: linux/amd64
platforms: linux/amd64,linux/arm64

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n## container-build workflow outline\n'
ast-grep outline .github/workflows/container-build.yml --view expanded || true

printf '\n## container-build workflow with line numbers (selected ranges)\n'
nl -ba .github/workflows/container-build.yml | sed -n '1,220p'

printf '\n## search for timeout, qemu, buildx, arm64, and cache-related settings in workflow docs\n'
rg -n --hidden -S 'timeout-minutes|qemu|buildx|arm64|platforms:|cache|cuda' .github/workflows .github README.md CONTRIBUTING.md mkdocs.yml docs src || true

printf '\n## list workflow files for context\n'
git ls-files .github/workflows

Repository: NVIDIA-NeMo/Safe-Synthesizer

Length of output: 360


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n## line count\n'
wc -l .github/workflows/container-build.yml

printf '\n## selected workflow ranges\n'
awk 'NR>=1 && NR<=220 { printf "%4d  %s\n", NR, $0 }' .github/workflows/container-build.yml

printf '\n## relevant search hits\n'
rg -n -S 'timeout-minutes|qemu|buildx|arm64|platforms:|cache|cuda' .github/workflows/container-build.yml .github/workflows || true

printf '\n## related docs mentions\n'
rg -n -S 'cache|CUDA|arm64|QEMU|timeout|buildx|platforms' .github README.md CONTRIBUTING.md mkdocs.yml docs src || true

Repository: NVIDIA-NeMo/Safe-Synthesizer

Length of output: 50384


Avoid emulating the arm64 CUDA leg
platforms: linux/amd64,linux/arm64 plus setup-qemu-action for arm64 runs the arm64 build under QEMU in this same job. For a CUDA Dockerfile, that makes the release path much slower and more fragile; split amd64/arm64 into separate jobs or move arm64 to a native runner.

Comment thread docs/developer-guide/docker.md Outdated
Comment thread .github/workflows/container-build.yml
Comment thread .github/workflows/container-build.yml
Signed-off-by: Matt Kornfield <mkornfield@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:build-dist area:ci area:docs chore Maintenance not tied to a user-visible change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants