build: generate CUDA dependency metadata from one source#655
build: generate CUDA dependency metadata from one source#655binaryaaron wants to merge 4 commits into
Conversation
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
github-actions
Bot
added
area:dev-ex
WalkthroughChangesThe PR adds CUDA Metadata Generation
Estimated code review effort: 4 (Complex) | ~60 minutes Suggested labels: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| return spec.as_pepstr(self) | ||
| raise TypeError(f"Unsupported dependency entry {dependency!r}") | ||
|
|
||
| def applies(self, dependency: DependencyEntry) -> bool: |
coderabbitai
Bot
added
feature
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Greptile SummaryThis PR moves the CPU and CUDA runtime extras, uv sources, and package indexes out of hand-maintained
Confidence Score: 5/5Safe to merge; the generator is fully deterministic, the generated pyproject.toml sections are consistent with cuda_deps.toml, and the previous thread issues are all resolved in this revision. The core generation logic is correct and well-tested: idempotency, stale-marker replacement, drift detection, and a dozen error-path validations are all covered. The pyproject.toml changes are purely reformatting of existing entries plus the addition of more precise platform markers on torchao and triton sources, which are stricter and correct. No functional regressions were identified. No files require special attention; the two inline comments flag minor future-proofing concerns in tools/gen_cuda_deps.py that do not affect current correctness. Important Files Changed
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| @dataclass(frozen=True) | ||
| class NvidiaCudaLibrarySpec(StrictModel): | ||
| """NVIDIA CUDA package source routing metadata.""" | ||
|
|
||
| name: str = Field(description="NVIDIA CUDA library package stem without the nvidia- prefix.") | ||
| nvidia_package_suffix: str | None = Field( | ||
| default=None, | ||
| description="Optional package suffix override. Defaults to the CUDA variant suffix.", | ||
| ) | ||
| index: str | None = Field( | ||
| default=None, | ||
| description="Optional literal uv index name or template. Defaults to the variant PyTorch index.", | ||
| ) |
There was a problem hiding this comment.
@dataclass(frozen=True) applied to a Pydantic BaseModel subclass
NvidiaCudaLibrarySpec combines Python's standard @dataclass(frozen=True) with Pydantic's BaseModel, which is non-standard. Because Pydantic v2 generates __init__ directly into each model class's __dict__, the @dataclass decorator skips its own __init__ — but it still writes __setattr__/__delattr__ that raise FrozenInstanceError, bypassing Pydantic's own immutability machinery. The correct approach for frozen Pydantic models is model_config = ConfigDict(frozen=True), which is already available through StrictModel's configuration. This combination may break silently across Pydantic minor releases if that internal __init__ placement changes.
| def _update_pyproject( | ||
| pyproject_path: Path, | ||
| check: bool, | ||
| generated: CudaPyprojectFragment, | ||
| ) -> GenerationResult: | ||
| current = pyproject_path.read_text(encoding="utf-8") | ||
| updated = apply_cuda_fragment_to_pyproject(current, generated) | ||
| if check: | ||
| return _check_pyproject(pyproject_path, current, updated) | ||
| pyproject_path.write_text(updated, encoding="utf-8") | ||
| return GenerationResult( | ||
| status=GenStatus.ok, | ||
| message=f"Updated generated CUDA dependency sections in {pyproject_path}", | ||
| ) |
There was a problem hiding this comment.
_update_pyproject unconditionally calls pyproject_path.write_text(updated, ...) and returns "Updated generated CUDA dependency sections" even when current == updated. This means running the generator when the file is already up-to-date still modifies the mtime, can create a spurious git diff, and produces a misleading success message. A pre-check if current == updated: return GenerationResult(status=GenStatus.ok, message="... already up to date") before the write would prevent this.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: b4bffca5-fd31-4a5a-8bfe-d35cea816d23
📒 Files selected for processing (7)
.agents/skills/uv-build/SKILL.md.mise/tasks/quality.tomlCONTRIBUTING.mdcuda_deps.tomlpyproject.tomltests/test_gen_cuda_deps.pytools/gen_cuda_deps.py
📜 Review details
⏰ Context from checks skipped due to timeout. (4)
- GitHub Check: Unit Tests (3.13)
- GitHub Check: Unit Tests (3.12)
- GitHub Check: Unit Tests (3.11)
- GitHub Check: Greptile Review
🧰 Additional context used
📓 Path-based instructions (17)
**/*.{md,markdown,py}
📄 CodeRabbit inference engine (.cursor/rules/agent-markdown-style.mdc)
**/*.{md,markdown,py}: Avoid decorative bold (**text**) in list items, body text, and docstrings; use structural cues (headers, list markers, colons, backticks) for emphasis instead
Use backticks for code identifiers, paths, and CLI commands in markdown and docstrings
Files:
CONTRIBUTING.mdtests/test_gen_cuda_deps.pytools/gen_cuda_deps.py
**/*.{md,markdown}
📄 CodeRabbit inference engine (.cursor/rules/agent-markdown-style.mdc)
**/*.{md,markdown}: Bold is acceptable only in markdown tables where it's the conventional way to mark header-like cells in the body
Use##headers to segment markdown sections instead of bold text
Use--(em-dash) instead of-(hyphen) for asides in markdown
Files:
CONTRIBUTING.md
**/*.md
📄 CodeRabbit inference engine (STYLE_GUIDE.md)
Do not use decorative bold in Markdown body text, list items, or docstrings; use single backticks for code identifiers, paths, and commands.
Use the repository's MkDocs Material Markdown conventions, including supported admonitions, content tabs, fenced code blocks, Mermaid diagrams, task lists, footnotes, definition lists, and emoji where appropriate.
Files:
CONTRIBUTING.md
**/*
📄 CodeRabbit inference engine (STYLE_GUIDE.md)
**/*: Every source file requires the SPDX copyright and license header appropriate to its file format.
End files with a newline, remove trailing whitespace, use one space between sentences, and keep code, comments, and docstrings within 120 characters.
**/*: Never move a published release tag; if release code changes, create and validate the next release candidate instead.
A stable release tag must point to the exact tested commit SHA, not a latermaincommit.
Before publishing a release, verify the wheel, package import, CLI, dependency set, GitHub release, PyPI artifacts, and immutable container image tag.
All contributions must include a DCOSigned-off-bytrailer, and commits must also have a verified cryptographic signature.
Commits merged tomainmust follow Conventional Commits syntax with a lowercase valid type, optional scope, description of at most 100 characters, and!for breaking changes.
Before submitting a pull request, all existing tests must pass; new features require tests and bug fixes require regression tests.
Use the repository'smisetasks for development, testing, formatting, checking, documentation, and release operations instead of deprecated Makefile task commands.
Files:
CONTRIBUTING.mdcuda_deps.tomlpyproject.tomltests/test_gen_cuda_deps.pytools/gen_cuda_deps.py
⚙️ CodeRabbit configuration file
**/*: Review as a senior maintainer for NeMo Safe Synthesizer. Prioritize issues that can change behavior, break user workflows, weaken privacy guarantees, hide failures, make tests unreliable, or create maintenance risk. Avoid generic style commentary unless it points to a concrete project convention that automated tools will not catch.
Comment only when the finding is actionable and tied to changed code. For each finding, state the impact, the condition that triggers it, and the smallest practical fix. Prefer one precise comment over broad advice. Do not ask for refactors outside the PR scope unless the changed code creates the problem.
Review type guidance: - Potential issue: use for correctness bugs, data loss, privacy leaks,
security risks, broken public APIs, invalid config behavior, missing
validation, hidden failures, nondeterministic tests, or CI breakage.
- Refactor suggestion: use for local maintainability problems introduced
by the diff when they have clear future cost, such as duplicated setup,
unclear boundaries, over-mocking, avoidable complexity, or opaque test
helpers.- Nitpick: avoid in chill mode. Do not emit formatting, import-order,
wording, or style-only comments unless automated tools cannot catch the
issue and it affects maintainability.Severity guidance: - Critical: security/privacy leaks, data loss, training/test/holdout
contamination, or broken release/package/core pipeline execution.
- Major: incorrect generation/training/evaluation behavior, broken
CLI/SDK public API, invalid config defaults or validators, or GPU/vLLM
cleanup and process-isolation bugs likely to fail CI or production
runs.- Minor: localized bugs, missing focused tests for changed behavior, or
bad test patterns that weaken regression coverage.- Trivial: small cleanup with no behavior impact. Usually suppress in
chill mode.- Info: context only. Avoid unless it helps reviewers understand risk.
Safe-Synthesizer-specific review focus: - Data ...
Files:
CONTRIBUTING.mdcuda_deps.tomlpyproject.tomltests/test_gen_cuda_deps.pytools/gen_cuda_deps.py
**/*.{py,sh,yaml,yml,md}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.{py,sh,yaml,yml,md}: All source files with.py,.sh,.yaml,.yml, or.mdextensions must include SPDX copyright headers.
Use the repository's pinnedmisetasks and formatting checks rather than relying on locally installed tool versions.
Files:
CONTRIBUTING.mdtests/test_gen_cuda_deps.pytools/gen_cuda_deps.py
.agents/skills/**
📄 CodeRabbit inference engine (.cursor/rules/repo-navigation.mdc)
Store skills in canonical location
.agents/skills/with each skill containing a SKILL.md file and optional references/
Files:
.agents/skills/uv-build/SKILL.md
**/*.toml
📄 CodeRabbit inference engine (STYLE_GUIDE.md)
Use spaces around
=, comment dependency pins inline, and follow the prescribedpyproject.tomlsection order.
Files:
cuda_deps.tomlpyproject.toml
cuda_deps.toml
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Changes to
cuda_deps.tomlrequire regenerated CUDA metadata and lockfile validation viamise run lock-check.
Files:
cuda_deps.toml
.mise/tasks/**/*.toml
📄 CodeRabbit inference engine (STYLE_GUIDE.md)
Keep declarative Mise tasks under
.mise/tasks/, provide descriptions for public tasks, and add usage metadata where arguments need validation or help.
Files:
.mise/tasks/quality.toml
.mise/tasks/*
📄 CodeRabbit inference engine (STYLE_GUIDE.md)
Keep shared shell helpers in
.mise/tasks/_lib.shand make that file non-executable.
Files:
.mise/tasks/quality.toml
pyproject.toml
📄 CodeRabbit inference engine (.cursor/rules/repo-navigation.mdc)
Configure package metadata, dependencies, extras (cpu/cu129/engine), and uv configuration in
pyproject.tomlKeep generated CUDA metadata and the
uvlockfile synchronized when changingpyproject.tomlorcuda_deps.toml; validate drift withmise run lock-check.
Files:
pyproject.toml
⚙️ CodeRabbit configuration file
Treat pyproject.toml as high-risk. Check package metadata, uv indexes, dependency groups, optional extras, Python version bounds, hatch config, ty config, script entry points, dependency consistency, and whether changes require regenerating uv.lock.
Files:
pyproject.toml
**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
**/*.py: Place durable implementation guidance in function and class docstrings for public contracts and source comments for local invariants
Target Python 3.11–3.13 with modern syntax (X | Y,list[str],Self). Python 3.14+ is not supported
**/*.py: Use American English spelling in Python code, documentation, and messages.
UseField(description=...)for every Pydantic model field.
Use assignment-styleField()by default; useAnnotatedonly for additional metadata such as validators, constrained aliases, or discriminated unions.
Use@dataclass(frozen=True)for immutable value objects and validators; use mutable dataclasses only for builders, accumulators, and pipeline state.
Usefield(default_factory=list)instead of mutable list defaults.
UseStrEnumfor string-valued configuration or serialization enums and plainEnumfor internal constants.
Obtain loggers withobservability.get_logger(__name__); do not calllogging.getLogger()orstructlog.get_logger()directly.
Use.runtime,.user, and.systemcategory loggers appropriately.
Do not useprint()for operational library output; use the approved logger,click.echo(), orsys.stdout.write()where appropriate.
Useextra={}for machine-queryable logging data and f-strings only for human-readable context.
Raise errors from the custom Safe Synthesizer error hierarchy, using the documented dual inheritance for user and internal errors.
Keep shared package code compatible with Python 3.11; do not use Python 3.12-only syntax such as PEP 695 type statements or bracketed generic parameters.
PreferX | Y, built-in collection generics, andSelfoverOptional,Union, and legacy typing collections.
Use collection ABCs for function arguments and concrete collection types for return values.
UseProtocolfor structural subtyping and avoidAnywhenobject, generics, or protocols are suitable.
UseTYPE_CHECKINGguards for heavy imports such as pandas, torch, and transformers.
...
Files:
tests/test_gen_cuda_deps.pytools/gen_cuda_deps.py
**/test_*.py
📄 CodeRabbit inference engine (AGENTS.md)
Use the
unitmarker instead of the deprecatedunit_testmarker for test identification
Files:
tests/test_gen_cuda_deps.py
tests/**
📄 CodeRabbit inference engine (.cursor/rules/repo-navigation.mdc)
tests/**: Mirrorsrc/directory structure intests/directory for test organization
Auto-mark tests by directory:tests/e2e/→e2e,tests/smoke/→smoke, otherwise default tounitMirror source code directory structure in tests directory (e.g.,
tests/training/,tests/generation/parallel to source structure)
Files:
tests/test_gen_cuda_deps.py
tests/**/*.py
📄 CodeRabbit inference engine (tests/TESTING.md)
tests/**/*.py: Auto-mark tests based on file path: tests under/e2e/gete2emarker, tests under/smoke/getsmokemarker, all others getunitmarker (only if no category marker already present)
Every test should have exactly one category marker:unit,smoke, ore2e
Usepytest.mark.requires_gpumodifier on tests that need CUDA hardware
Usepytest.mark.vllmon tests using vLLM generation backend and ensure each vLLM test file runs in its own process for GPU memory isolation
Usepytest.mark.slowon long-running tests
Usepytest.mark.smollm2for SmolLM2 Hub download tests to enable process isolation
Usepytest.mark.noautouseto skip autouse fixtures for specific tests
Useload_test_dataset(filename)helper to load test datasets fromtests/stub_datasets/as HuggingFaceDatasetobjects
Useload_test_dataframe(filename)helper to load test data files fromtests/stub_datasets/as pandas DataFrames
Convert pandas columns to nullable dtypes (pd.Int64Dtype(),pd.BooleanDtype()) before assigningnp.nanvalues
Usefake.seed_instance(seed)andrandom.seed(seed)together for Faker-based test data reproducibility
When sharing methods across multiple test files, define them inconftest.pyand import them using relative imports (e.g.,from .conftest import train_with_sdk); note that importing from other test files liketests/cli/helpers.pydoes not work
Usefixture_mock_processororfixture_mock_processor_without_valid_recordsfor mocking ParsedResponse objects withvalid_records,invalid_records,errors, andprompt_numberfields
Usepytest.importorskipto gate tests on optional dependencies that require specific extras (e.g.,sentence_transformers,vllm)
Run vLLM tests with separate pytest invocations (one per file) using-n 0(single process) for GPU memory isolation, or use staged mise tasks for CI visibility
Print statements are allowed in tests (ruffT201is suppressed fortests/directory) and should...
Files:
tests/test_gen_cuda_deps.py
⚙️ CodeRabbit configuration file
Review tests against tests/TESTING.md. Check marker usage, fixture naming, tmp_path usage, determinism, and GPU/vLLM process-isolation requirements. Flag slop tests that only check that code runs, assert result is not None when stronger invariants exist, over-mock internal implementation details, patch around the bug instead of reproducing it, or add broad snapshot/golden churn without a clear contract. Flag change detector tests that fail on harmless refactors, formatting, record ordering, incidental wording, or private implementation details without demonstrating a behavior regression. Prefer existing fixtures or focused new fixtures for repeated setup; keep tests DRY when reasonable without making the behavior under test opaque. print() is allowed in tests.
Files:
tests/test_gen_cuda_deps.py
tests/test_*.py
📄 CodeRabbit inference engine (STYLE_GUIDE.md)
Name test files
test_*.py, classesTest*, and functionstest_<module>_<expected_behavior>.
Files:
tests/test_gen_cuda_deps.py
tools/**
⚙️ CodeRabbit configuration file
Review tools as developer and CI infrastructure. Check that scripts use uv or Makefile wrappers instead of ad hoc python/pip commands, preserve read-only behavior for check targets, fail with clear messages, avoid hidden network or filesystem side effects, and stay consistent with STYLE_GUIDE.md and CONTRIBUTING.md. Tooling may use print() when it is a standalone script or intentional CLI output.
Files:
tools/gen_cuda_deps.py
🧠 Learnings (1)
📚 Learning: 2026-05-27T22:20:37.354Z
Learnt from: kendrickb-nvidia
Repo: NVIDIA-NeMo/Safe-Synthesizer PR: 520
File: tests/generation/test_vllm_backend.py:556-587
Timestamp: 2026-05-27T22:20:37.354Z
Learning: In NVIDIA-NeMo/Safe-Synthesizer, `tests/conftest.py`’s `pytest_collection_modifyitems` hook applies pytest category markers automatically based on each test file’s path: tests under `/e2e/` get `pytest.mark.e2e`, tests under `/smoke/` get `pytest.mark.smoke`, and all other tests get `pytest.mark.unit`. Therefore, when reviewing pytest tests outside `tests/e2e/` and `tests/smoke/`, do not flag missing explicit `pytest.mark.unit` decorators on test classes/functions as an issue (the hook will add them during collection). If a new test directory/category is introduced, ensure the hook is updated so it’s categorized correctly.
Applied to files:
tests/test_gen_cuda_deps.py
🪛 LanguageTool
.agents/skills/uv-build/SKILL.md
[grammar] ~70-~70: Use a hyphen to join words.
Context: ...ock`. Pre-commit verifies the lock is up to date. The generated CPU/CUDA sections of...
(QB_NEW_EN_HYPHEN)
🔇 Additional comments (7)
cuda_deps.toml (1)
1-59: LGTM!Also applies to: 64-112
tools/gen_cuda_deps.py (1)
1-4: LGTM!Also applies to: 6-13, 22-223, 238-983
pyproject.toml (1)
113-244: LGTM!Also applies to: 252-331
tests/test_gen_cuda_deps.py (1)
1-3: LGTM!Also applies to: 9-149, 155-340
.mise/tasks/quality.toml (1)
24-25: LGTM!Also applies to: 27-29
.agents/skills/uv-build/SKILL.md (1)
64-74: LGTM!CONTRIBUTING.md (1)
596-596: LGTM!
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| # /// script | ||
| # requires-python = ">=3.11" |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '--- tools/gen_cuda_deps.py ---\n'
sed -n '1,220p' tools/gen_cuda_deps.py
printf '\n--- version constraints ---\n'
rg -n --hidden --glob '!**/.git/**' 'requires-python|3\.14|3\.13|3\.11' pyproject.toml tools README.md CONTRIBUTING.md STYLE_GUIDE.md .github || trueRepository: NVIDIA-NeMo/Safe-Synthesizer
Length of output: 257
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '1,220p' tools/gen_cuda_deps.py
printf '\n---\n'
rg -n --hidden --glob '!**/.git/**' '^requires-python|python = "' pyproject.toml tools/*.py .github CONTRIBUTING.md STYLE_GUIDE.md README.md || trueRepository: NVIDIA-NeMo/Safe-Synthesizer
Length of output: 9211
🏁 Script executed:
#!/bin/bash
set -euo pipefail
nl -ba tools/gen_cuda_deps.py | sed -n '1,120p'Repository: NVIDIA-NeMo/Safe-Synthesizer
Length of output: 206
Cap tools/gen_cuda_deps.py at Python 3.13.
tools/gen_cuda_deps.py:5 is a standalone uv run --script entry, so requires-python = ">=3.11" still allows 3.14+. Tighten it to >=3.11,<3.14 to match the supported runtime window.
Source: Coding guidelines
| """Generate CPU and CUDA dependency metadata in pyproject.toml.""" | ||
|
|
||
| import tomllib |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '--- tools/gen_cuda_deps.py ---\n'
cat -n tools/gen_cuda_deps.py | sed -n '1,40p'
printf '\n--- tests/test_gen_cuda_deps.py ---\n'
cat -n tests/test_gen_cuda_deps.py | sed -n '1,40p'Repository: NVIDIA-NeMo/Safe-Synthesizer
Length of output: 257
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- tools/gen_cuda_deps.py ---'
sed -n '1,40p' tools/gen_cuda_deps.py | cat -n
echo
echo '--- tests/test_gen_cuda_deps.py ---'
sed -n '1,40p' tests/test_gen_cuda_deps.py | cat -nRepository: NVIDIA-NeMo/Safe-Synthesizer
Length of output: 3126
Add from __future__ import annotations to both new modules.
tools/gen_cuda_deps.pytests/test_gen_cuda_deps.py
📍 Affects 2 files
tools/gen_cuda_deps.py#L14-L16(this comment)tests/test_gen_cuda_deps.py#L4-L7
Source: Coding guidelines
Addresses PR #655 review comments: scope CPU torch/torchaudio/torchvision sources to Linux, skip rewriting an unchanged pyproject.toml, drop unused structlog dependency and dead GenStatus.error, use Pydantic's frozen model config instead of mixing in @DataClass, match the script's Python range to the repo, and run the lock-check generator call offline. Also simplifies tools/gen_cuda_deps.py: drop the PEP 723 inline-script metadata block in favor of running against the repo venv, fix a marker duplication bug in the generated-block stripping logic (surfaced while dropping that block), fold single-caller helpers into their only callers (_render_template, GeneratedBlocks' position finders), derive the uv source marker from sys_platform/arch instead of hand-duplicating it in cuda_deps.toml, and move CudaVariantUvRouter's index-resolution methods onto CudaVariantContext to remove multi-hop reaches through it. Reworks the test fixtures to use structural TOML mutation instead of brittle string-replace against a shared blob, and extracts duplicated expected-output constants. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
There was a problem hiding this comment.
Actionable comments posted: 4
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 16e068a0-3deb-47ca-bcc3-03870e2efcf0
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock,!uv.lock
📒 Files selected for processing (5)
.mise/tasks/quality.tomlcuda_deps.tomlpyproject.tomltests/test_gen_cuda_deps.pytools/gen_cuda_deps.py
🚧 Files skipped from review as they are similar to previous changes (3)
- .mise/tasks/quality.toml
- pyproject.toml
- cuda_deps.toml
📜 Review details
⏰ Context from checks skipped due to timeout. (8)
- GitHub Check: Unit Tests (3.12)
- GitHub Check: Unit Tests (3.11)
- GitHub Check: Unit Tests (3.13)
- GitHub Check: Smoke Tests
- GitHub Check: End-user Wheel Install
- GitHub Check: Greptile Review
- GitHub Check: Analyze (Python)
- GitHub Check: Analyze (python)
🧰 Additional context used
📓 Path-based instructions (9)
**/*.{md,markdown,py}
📄 CodeRabbit inference engine (.cursor/rules/agent-markdown-style.mdc)
**/*.{md,markdown,py}: Avoid decorative bold (**text**) in list items, body text, and docstrings; use structural cues (headers, list markers, colons, backticks) for emphasis instead
Use backticks for code identifiers, paths, and CLI commands in markdown and docstrings
Files:
tests/test_gen_cuda_deps.pytools/gen_cuda_deps.py
**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
**/*.py: Place durable implementation guidance in function and class docstrings for public contracts and source comments for local invariants
Target Python 3.11–3.13 with modern syntax (X | Y,list[str],Self). Python 3.14+ is not supported
**/*.py: Use American English spelling in Python code, documentation, and messages.
UseField(description=...)for every Pydantic model field.
Use assignment-styleField()by default; useAnnotatedonly for additional metadata such as validators, constrained aliases, or discriminated unions.
Use@dataclass(frozen=True)for immutable value objects and validators; use mutable dataclasses only for builders, accumulators, and pipeline state.
Usefield(default_factory=list)instead of mutable list defaults.
UseStrEnumfor string-valued configuration or serialization enums and plainEnumfor internal constants.
Obtain loggers withobservability.get_logger(__name__); do not calllogging.getLogger()orstructlog.get_logger()directly.
Use.runtime,.user, and.systemcategory loggers appropriately.
Do not useprint()for operational library output; use the approved logger,click.echo(), orsys.stdout.write()where appropriate.
Useextra={}for machine-queryable logging data and f-strings only for human-readable context.
Raise errors from the custom Safe Synthesizer error hierarchy, using the documented dual inheritance for user and internal errors.
Keep shared package code compatible with Python 3.11; do not use Python 3.12-only syntax such as PEP 695 type statements or bracketed generic parameters.
PreferX | Y, built-in collection generics, andSelfoverOptional,Union, and legacy typing collections.
Use collection ABCs for function arguments and concrete collection types for return values.
UseProtocolfor structural subtyping and avoidAnywhenobject, generics, or protocols are suitable.
UseTYPE_CHECKINGguards for heavy imports such as pandas, torch, and transformers.
...
Files:
tests/test_gen_cuda_deps.pytools/gen_cuda_deps.py
**/test_*.py
📄 CodeRabbit inference engine (AGENTS.md)
Use the
unitmarker instead of the deprecatedunit_testmarker for test identification
Files:
tests/test_gen_cuda_deps.py
tests/**
📄 CodeRabbit inference engine (.cursor/rules/repo-navigation.mdc)
tests/**: Mirrorsrc/directory structure intests/directory for test organization
Auto-mark tests by directory:tests/e2e/→e2e,tests/smoke/→smoke, otherwise default tounitMirror source code directory structure in tests directory (e.g.,
tests/training/,tests/generation/parallel to source structure)
Files:
tests/test_gen_cuda_deps.py
tests/**/*.py
📄 CodeRabbit inference engine (tests/TESTING.md)
tests/**/*.py: Auto-mark tests based on file path: tests under/e2e/gete2emarker, tests under/smoke/getsmokemarker, all others getunitmarker (only if no category marker already present)
Every test should have exactly one category marker:unit,smoke, ore2e
Usepytest.mark.requires_gpumodifier on tests that need CUDA hardware
Usepytest.mark.vllmon tests using vLLM generation backend and ensure each vLLM test file runs in its own process for GPU memory isolation
Usepytest.mark.slowon long-running tests
Usepytest.mark.smollm2for SmolLM2 Hub download tests to enable process isolation
Usepytest.mark.noautouseto skip autouse fixtures for specific tests
Useload_test_dataset(filename)helper to load test datasets fromtests/stub_datasets/as HuggingFaceDatasetobjects
Useload_test_dataframe(filename)helper to load test data files fromtests/stub_datasets/as pandas DataFrames
Convert pandas columns to nullable dtypes (pd.Int64Dtype(),pd.BooleanDtype()) before assigningnp.nanvalues
Usefake.seed_instance(seed)andrandom.seed(seed)together for Faker-based test data reproducibility
When sharing methods across multiple test files, define them inconftest.pyand import them using relative imports (e.g.,from .conftest import train_with_sdk); note that importing from other test files liketests/cli/helpers.pydoes not work
Usefixture_mock_processororfixture_mock_processor_without_valid_recordsfor mocking ParsedResponse objects withvalid_records,invalid_records,errors, andprompt_numberfields
Usepytest.importorskipto gate tests on optional dependencies that require specific extras (e.g.,sentence_transformers,vllm)
Run vLLM tests with separate pytest invocations (one per file) using-n 0(single process) for GPU memory isolation, or use staged mise tasks for CI visibility
Print statements are allowed in tests (ruffT201is suppressed fortests/directory) and should...
Files:
tests/test_gen_cuda_deps.py
⚙️ CodeRabbit configuration file
Review tests against tests/TESTING.md. Check marker usage, fixture naming, tmp_path usage, determinism, and GPU/vLLM process-isolation requirements. Flag slop tests that only check that code runs, assert result is not None when stronger invariants exist, over-mock internal implementation details, patch around the bug instead of reproducing it, or add broad snapshot/golden churn without a clear contract. Flag change detector tests that fail on harmless refactors, formatting, record ordering, incidental wording, or private implementation details without demonstrating a behavior regression. Prefer existing fixtures or focused new fixtures for repeated setup; keep tests DRY when reasonable without making the behavior under test opaque. print() is allowed in tests.
Files:
tests/test_gen_cuda_deps.py
tests/test_*.py
📄 CodeRabbit inference engine (STYLE_GUIDE.md)
Name test files
test_*.py, classesTest*, and functionstest_<module>_<expected_behavior>.
Files:
tests/test_gen_cuda_deps.py
**/*
📄 CodeRabbit inference engine (STYLE_GUIDE.md)
**/*: Every source file requires the SPDX copyright and license header appropriate to its file format.
End files with a newline, remove trailing whitespace, use one space between sentences, and keep code, comments, and docstrings within 120 characters.
**/*: Commits merged tomainmust use Conventional Commits format:<type>(<scope>): <description>or<type>: <description>, with a valid lowercase type and a description of at most 100 characters.
All contributions must include a DCOSigned-off-bytrailer, and commits must also have a verified cryptographic signature.
Branches other thanmainmust follow the lowercase<author>/<description>convention, optionally including an issue ID and one of the approved type prefixes.
Before submitting a pull request, run the repository's formatting, quality checks, and tests (mise run format,mise run check, andmise run test).
Files:
tests/test_gen_cuda_deps.pytools/gen_cuda_deps.py
⚙️ CodeRabbit configuration file
**/*: Review as a senior maintainer for NeMo Safe Synthesizer. Prioritize issues that can change behavior, break user workflows, weaken privacy guarantees, hide failures, make tests unreliable, or create maintenance risk. Avoid generic style commentary unless it points to a concrete project convention that automated tools will not catch.
Comment only when the finding is actionable and tied to changed code. For each finding, state the impact, the condition that triggers it, and the smallest practical fix. Prefer one precise comment over broad advice. Do not ask for refactors outside the PR scope unless the changed code creates the problem.
Review type guidance: - Potential issue: use for correctness bugs, data loss, privacy leaks,
security risks, broken public APIs, invalid config behavior, missing
validation, hidden failures, nondeterministic tests, or CI breakage.
- Refactor suggestion: use for local maintainability problems introduced
by the diff when they have clear future cost, such as duplicated setup,
unclear boundaries, over-mocking, avoidable complexity, or opaque test
helpers.- Nitpick: avoid in chill mode. Do not emit formatting, import-order,
wording, or style-only comments unless automated tools cannot catch the
issue and it affects maintainability.Severity guidance: - Critical: security/privacy leaks, data loss, training/test/holdout
contamination, or broken release/package/core pipeline execution.
- Major: incorrect generation/training/evaluation behavior, broken
CLI/SDK public API, invalid config defaults or validators, or GPU/vLLM
cleanup and process-isolation bugs likely to fail CI or production
runs.- Minor: localized bugs, missing focused tests for changed behavior, or
bad test patterns that weaken regression coverage.- Trivial: small cleanup with no behavior impact. Usually suppress in
chill mode.- Info: context only. Avoid unless it helps reviewers understand risk.
Safe-Synthesizer-specific review focus: - Data ...
Files:
tests/test_gen_cuda_deps.pytools/gen_cuda_deps.py
**/*.{py,sh,yaml,yml,md}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
All Python, shell, YAML, YML, and Markdown source files must include SPDX copyright headers, except files listed in
.copyrightignore.
Files:
tests/test_gen_cuda_deps.pytools/gen_cuda_deps.py
tools/**
⚙️ CodeRabbit configuration file
Review tools as developer and CI infrastructure. Check that scripts use uv or Makefile wrappers instead of ad hoc python/pip commands, preserve read-only behavior for check targets, fail with clear messages, avoid hidden network or filesystem side effects, and stay consistent with STYLE_GUIDE.md and CONTRIBUTING.md. Tooling may use print() when it is a standalone script or intentional CLI output.
Files:
tools/gen_cuda_deps.py
🧠 Learnings (1)
📚 Learning: 2026-05-27T22:20:37.354Z
Learnt from: kendrickb-nvidia
Repo: NVIDIA-NeMo/Safe-Synthesizer PR: 520
File: tests/generation/test_vllm_backend.py:556-587
Timestamp: 2026-05-27T22:20:37.354Z
Learning: In NVIDIA-NeMo/Safe-Synthesizer, `tests/conftest.py`’s `pytest_collection_modifyitems` hook applies pytest category markers automatically based on each test file’s path: tests under `/e2e/` get `pytest.mark.e2e`, tests under `/smoke/` get `pytest.mark.smoke`, and all other tests get `pytest.mark.unit`. Therefore, when reviewing pytest tests outside `tests/e2e/` and `tests/smoke/`, do not flag missing explicit `pytest.mark.unit` decorators on test classes/functions as an issue (the hook will add them during collection). If a new test directory/category is introduced, ensure the hook is updated so it’s categorized correctly.
Applied to files:
tests/test_gen_cuda_deps.py
🔇 Additional comments (2)
tools/gen_cuda_deps.py (1)
1-42: LGTM!Also applies to: 171-223, 252-540, 761-783, 899-937
tests/test_gen_cuda_deps.py (1)
33-204: LGTM!Also applies to: 235-369
| def test_apply_cuda_fragment_to_pyproject_replaces_stale_marker_text(tmp_path: Path, generator: ModuleType) -> None: | ||
| """Regenerating over a header written by an older script version must replace it, not leave a stray copy.""" | ||
| config_path = tmp_path / "cuda_deps.toml" | ||
| config_path.write_text(CUDA_DEPS, encoding="utf-8") | ||
| generated = generator.build_cuda_pyproject_fragment(generator.load_cuda_deps_config(config_path)) | ||
| updated = generator.apply_cuda_fragment_to_pyproject(PYPROJECT, generated) | ||
|
|
||
| stale = updated.replace( | ||
| "# Regenerate with: uv run --frozen tools/gen_cuda_deps.py cuda_deps.toml --pyproject pyproject.toml", | ||
| "# Regenerate with: uv run --script tools/gen_cuda_deps.py cuda_deps.toml --pyproject pyproject.toml", | ||
| ) | ||
| assert stale != updated | ||
|
|
||
| reapplied = generator.apply_cuda_fragment_to_pyproject(stale, generated) | ||
|
|
||
| assert reapplied == updated | ||
| assert reapplied.count("Regenerate with") == 3 |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Exercise a stale header with a different line count.
This replacement preserves the current header shape, so it cannot detect the fixed-length stripping bug. Add or remove a header comment line before reapplying the fragment and assert exact convergence.
| runtime_start = self._find_assignment(self.extras[0]) | ||
| conflicts_start = self._find_assignment("conflicts") | ||
| yield GeneratedBlock( | ||
| label="RUNTIME EXTRAS", | ||
| start=runtime_start, | ||
| end=self._find_array_end(self._find_assignment(self.extras[-1])), | ||
| detail=f"# Generated extras in this block: {', '.join(self.extras)}.", | ||
| ) | ||
| yield GeneratedBlock( | ||
| label="UV CONFLICTS", | ||
| start=conflicts_start, | ||
| end=self._find_array_end(conflicts_start), | ||
| detail="# Generated uv section: tool.uv.conflicts.", | ||
| ) | ||
| yield GeneratedBlock( | ||
| label="UV SOURCES AND INDEXES", | ||
| start=self._find_section("[tool.uv.sources]"), | ||
| end=self._last_generated_index_line(), | ||
| detail="# Generated uv sections: tool.uv.sources and tool.uv.index.", | ||
| ) | ||
|
|
||
| def reversed(self) -> list[GeneratedBlock]: | ||
| return sorted(self, key=lambda block: block.start, reverse=True) | ||
|
|
||
| def _find_assignment(self, key: str) -> int: | ||
| assignment = f"{key} = [" | ||
| for index, line in enumerate(self.lines): | ||
| if line == assignment: | ||
| return index | ||
| raise ValueError(f"pyproject.toml: missing generated assignment {assignment!r}") | ||
|
|
||
| def _find_section(self, section: str) -> int: | ||
| for index, line in enumerate(self.lines): | ||
| if line == section: | ||
| return index | ||
| raise ValueError(f"pyproject.toml: missing generated section {section!r}") | ||
|
|
||
| def _last_generated_index_line(self) -> int: | ||
| index_start = self._find_section("[[tool.uv.index]]") | ||
| next_section = self._find_next_non_index_section(index_start + 1) | ||
| return next_section - 1 if next_section is not None else len(self.lines) - 1 | ||
|
|
||
| def _find_next_non_index_section(self, start_index: int) -> int | None: | ||
| for index in range(start_index, len(self.lines)): | ||
| if self.lines[index].startswith("[") and self.lines[index] != "[[tool.uv.index]]": | ||
| return index | ||
| return None | ||
|
|
||
| def _find_array_end(self, start_index: int) -> int: | ||
| for index in range(start_index + 1, len(self.lines)): | ||
| if self.lines[index] == "]": | ||
| return index | ||
| raise ValueError(f"pyproject.toml: missing closing bracket for generated assignment at line {start_index + 1}") |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Scope assignment searches to their owning TOML tables.
_find_assignment() scans the entire document, so an earlier unrelated cpu = [ or conflicts = [ is selected. Marker insertion—and later removal—can then modify unrelated configuration. Restrict searches to [project.optional-dependencies] and [tool.uv], with a collision regression test.
| def effective_source_marker(self, renderer: "TemplateRenderer") -> str | None: | ||
| """Marker for the generated uv source entry: an explicit override, else the requirement's own markers.""" | ||
| if self.source_marker is not None: | ||
| return renderer.template(self.source_marker) | ||
| markers = list(self.pep_markers(renderer)) | ||
| return " and ".join(markers) if markers else None |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Validate the rendered source marker before returning it.
Templated source_marker values bypass validation at Lines 201–203, and this method returns the rendered value directly. An invalid rendered marker therefore reaches generated tool.uv.sources. Parse it with Marker(rendered) before returning.
Proposed fix
if self.source_marker is not None:
- return renderer.template(self.source_marker)
+ rendered = renderer.template(self.source_marker)
+ Marker(rendered)
+ return renderedAs per path instructions, invalid configuration and missing validation must be treated as potential issues.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| def effective_source_marker(self, renderer: "TemplateRenderer") -> str | None: | |
| """Marker for the generated uv source entry: an explicit override, else the requirement's own markers.""" | |
| if self.source_marker is not None: | |
| return renderer.template(self.source_marker) | |
| markers = list(self.pep_markers(renderer)) | |
| return " and ".join(markers) if markers else None | |
| def effective_source_marker(self, renderer: "TemplateRenderer") -> str | None: | |
| """Marker for the generated uv source entry: an explicit override, else the requirement's own markers.""" | |
| if self.source_marker is not None: | |
| rendered = renderer.template(self.source_marker) | |
| Marker(rendered) | |
| return rendered | |
| markers = list(self.pep_markers(renderer)) | |
| return " and ".join(markers) if markers else None |
Source: Path instructions
| def _strip_generated_markers(lines: list[str]) -> list[str]: | ||
| # Header length is fixed by _insert_generated_marker (GENERATED_MARKER_BODY plus one | ||
| # detail line), so skip by position rather than matching comment text: matching text | ||
| # would silently miss stale headers left over from a previous version of that text. | ||
| header_length = len(GENERATED_MARKER_BODY) + 1 | ||
| stripped = [] | ||
| skip_remaining = 0 | ||
| for line in lines: | ||
| if skip_remaining: | ||
| skip_remaining -= 1 | ||
| continue | ||
| if line.startswith(GENERATED_BEGIN_PREFIX): | ||
| skip_remaining = header_length | ||
| continue | ||
| if line.startswith(GENERATED_END_PREFIX): | ||
| continue | ||
| stripped.append(line) | ||
| return stripped |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Handle stale generated headers without assuming their historical length.
tools/gen_cuda_deps.py#L702-L719: strip the consecutive header comments instead of skipping the current fixed count.tests/test_gen_cuda_deps.py#L216-L232: add a stale header with an extra or missing comment line and assert exact convergence.
📍 Affects 2 files
tools/gen_cuda_deps.py#L702-L719(this comment)tests/test_gen_cuda_deps.py#L216-L232
…tion pyproject.toml's CPU/CUDA sections are now generated from cuda_deps.toml, but AGENTS.md never mentioned this, docs/developer-guide/docker.md still told contributors to hand-edit pyproject.toml for a new variant, and .agents/skills/uv-build/SKILL.md contradicted its own generated-section warning with a stale "edit extras manually in pyproject.toml" convention and a --script invocation that no longer works now that the generator dropped its inline PEP 723 metadata. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
1 participant
Summary
this is one of three stacked prs (#656, #657) for our overdue multiple-versions-of-cuda support. This one is the core mechanism for generating the deps all from one place and handles instructions and docs for new installation methods.
Validation
Summary by CodeRabbit
New Features
Documentation
Quality & Tests