Repository navigation
feat(docs,examples): migrate tutorials and notebooks to nemotron-3.5-content-safety (NGUARD-972, NGUARD-980) - #2442
Open
svenchilton wants to merge 3 commits into
Conversation
…content-safety (NGUARD-980) + add benchmark eval script (NGUARD-972) Tutorials (docs/getting-started/tutorials/): - nemotron-safety-guard-deployment.mdx: migrated to nemotron-3.5-content-safety NIM with request_categories and chat_template_kwargs - nemotron-content-safety-reasoning-deployment.mdx: full rewrite for enable_thinking: true (reasoning/extended-thinking mode) - nemoguard-topiccontrol-deployment.mdx: full rewrite for custom_policy - multimodal.mdx: updated model ID and config for vision rail Notebooks (examples/notebooks/): - content_safety_nim.ipynb: updated to nemotron-3.5-content-safety NIM - topic_control_nim.ipynb: updated for custom_policy - combined_guardrails_nim.ipynb: updated model references; architecture under review -- custom_policy and request_categories are not additive in a single model call; correct fix requires two separate model instances (pending Tim Gasser / Anuj Doshi reply on NGUARD-980) Vision config (examples/configs/content_safety_vision/): - Migrated from gpt-4o-mini vision_rails to nemotron-3.5-content-safety NIM Benchmark eval script (examples/e2e_tests/): - test_content_safety_rail.py: end-to-end eval covering Aegis 2.0, Nemotron Safety Guard v3, CoSApien, DynaGuardrail, and VLGuard datasets against both direct NIM and Guardrails input rail (NGUARD-972) Signed-off-by: Sven Chilton <schilton@nvidia.com>
…enRouter note (NGUARD-980) - test_content_safety_rail.py: add --main-nim-url, --main-model-name, --guardrails-concurrency flags and stratified subsampling via --sample - gliner_pii_detection_nim.ipynb: add deprecation note for nvidia/gliner-pii hosted endpoint - combined_guardrails_nim.ipynb: note that OpenRouter-hosted GLiNER equivalent is coming soon - uv.lock: sync to current dependency resolution Signed-off-by: Sven Chilton <schilton@nvidia.com>
github-actions
Bot
added
status: needs triage
Contributor
|
Staged Fern docs preview: https://nvidia-preview-pr-2442.docs.buildwithfern.com/nemo/guardrails |
…y model The vision rail prompt was renamed from $model=vision_rails to $model=content_safety in examples/configs/content_safety_vision/prompts.yml as part of the nemotron-3.5 migration, but the test file still referenced the old name, causing all 13 CI jobs to fail with: ValueError: Could not find prompt for task content_safety_check_input $model=vision_rails and model openai/gpt-4o-mini Signed-off-by: Sven Chilton <schilton@nvidia.com>
Contributor
PR merge guidance@svenchilton thanks for the PR. GitHub is currently blocking merge for one or more repository requirements:
Relevant guide: |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
nemotron-3.5-content-safety(replacingnemotron-3-content-safety):nemotron-safety-guard-deployment.mdx,nemotron-content-safety-reasoning-deployment.mdx(full rewrite withenable_thinking: true),nemoguard-topiccontrol-deployment.mdx(full rewrite withcustom_policy),multimodal.mdx,topic_control_nim.ipynb,content_safety_nim.ipynb,combined_guardrails_nim.ipynb,gliner_pii_detection_nim.ipynbexamples/e2e_tests/test_content_safety_rail.py: end-to-end eval script covering Aegis 2.0, Nemotron Safety Guard v3, CoSApien, DynaGuardrail, and VLGuard; supports direct NIM path and full Guardrails pipeline comparison (--compare-guardrails); stratified subsampling (--sample), custom policy per dataset, per-domain/trickiness breakdowns (NGUARD-972, NGUARD-980)gliner_pii_detection_nim.ipynbandcombined_guardrails_nim.ipynb: hostednvidia/gliner-piiendpoint is no longer available on build.nvidia.com; notes local hosting as the current option and mentions an OpenRouter-hosted equivalent coming sooncombined_guardrails_nim.ipynbRemote Deployment sectioncontent_safety_visionconfig:config.ymlandprompts.ymlmigrated tonemotron-3.5-content-safetyuv.lock: regenerated with uv 0.11.26 to resolve stale lockfileEval results (direct NIM, default policy,
test_content_safety_rail.py)Guardrails pipeline (default policy) tracks direct within ≤0.004 F1 on all datasets (96.7–98.0% agreement), with ~2.7–2.8× latency overhead.
Test plan
test_content_safety_rail.pyruns against a live Nemotron 3.5 CS NIM (--dataset aegis --sample 10)Closes NGUARD-972, NGUARD-980