chore: trigger NVSkills CI for cuopt-install - #1672
Open
ramakrishnap-nv wants to merge 5 commits into
Open
Conversation
Collaborator
Author
|
/nvskills-ci |
📝 WalkthroughWalkthroughThe cuOpt install skill refresh updates its benchmark report, skill-card metadata, documentation spacing, and Sigstore signature bundle. The benchmark now records regenerated evaluation results, validation findings, scoring rules, and freshness criteria. ChangescuOpt install skill refresh
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
ramakrishnap-nv
added
improvement
CI Test Summary⏭️ All 5 test job(s) skipped. |
Collaborator
Author
|
/nvskills-ci |
Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@skills/cuopt-install/BENCHMARK.md`:
- Around line 74-87: Update the benchmark scoring documentation near the
dimension bands and overall verdict to define Overall as the arithmetic mean of
the five skill-assisted dimension scores, including the rounding rule and score
basis. State that the verdict gate uses these skill-assisted scores rather than
baseline values, while preserving the existing threshold and verdict rules.
- Around line 15-18: Update the benchmark report metadata in BENCHMARK.md to
identify the exact evaluated input file and task ID, specifically distinguishing
the Tier 3 dataset from the inst-eval-001-docker-server task. Reconcile the
Dataset digest and task count with the actual inputs evaluated, recording both
dataset details if both sources were used.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 71a63a7f-8e5b-40c8-83f9-3f5aaab8ee84
📒 Files selected for processing (3)
skills/cuopt-install/BENCHMARK.mdskills/cuopt-install/skill-card.mdskills/cuopt-install/skill.oms.sig
Comment on lines
+15
to
+18
| - Tasks: 1 evaluation tasks (1 positive) | ||
| - Dataset digest: `sha256:75bb16e95e852c956da6f5af155ac6936922740ae833f71436ddcb715da4b6df` (skill-evaluator-dataset-snapshot/1) | ||
| - Attempts per task: 1 | ||
| - Pass threshold: 50% | ||
| - Overall verdict: PASS | ||
| - Environment: `k8s-sandbox` |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
for name in (
"skills/cuopt-install/benchmark/evals.json",
"skills/cuopt-install/evals/evals.json",
):
value = json.loads(Path(name).read_text())
records = value if isinstance(value, list) else [value]
print(name)
for record in records:
if isinstance(record, dict):
print(" id:", record.get("id"))
PY
rg -n -C 3 \
-e 'install-001-required-questions|inst-eval-001-docker-server|benchmark/evals\.json|evals/evals\.json|Tasks:|evaluation task' \
skills/cuopt-install/BENCHMARK.md \
skills/cuopt-install/skill-card.mdRepository: NVIDIA/cuopt
Length of output: 1958
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- BENCHMARK.md ---'
cat -n skills/cuopt-install/BENCHMARK.md
printf '%s\n' '--- benchmark/evals.json records ---'
python3 - <<'PY'
import json
from pathlib import Path
for name in (
"skills/cuopt-install/benchmark/evals.json",
"skills/cuopt-install/evals/evals.json",
):
print(f"\n{name}")
value = json.loads(Path(name).read_text())
records = value if isinstance(value, list) else [value]
for record in records:
print(json.dumps(record, sort_keys=True, indent=2))
PY
printf '%s\n' '--- related benchmark metadata and references ---'
rg -n -C 4 \
-e '75bb16e|skill-evaluator-dataset-snapshot|install-001-required-questions|inst-eval-001-docker-server|benchmark/evals\.json|evals/evals\.json|dataset|digest|Tier 3|Overall|Freshness' \
skills/cuopt-install .github 2>/dev/null || trueRepository: NVIDIA/cuopt
Length of output: 29978
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- skill-card evaluation section ---'
cat -n skills/cuopt-install/skill-card.md | sed -n '40,90p'
printf '%s\n' '--- repository references to the two evaluation inputs ---'
rg -n -C 5 \
-e 'skills/cuopt-install/(benchmark/evals\.json|evals/evals\.json)' \
-e 'benchmark/evals\.json|evals/evals\.json' \
-e 'inst-eval-001-docker-server|install-001-required-questions' \
--glob '!skills/cuopt-install/benchmark/evals.json' \
--glob '!skills/cuopt-install/evals/evals.json' \
.
printf '%s\n' '--- candidate dataset digests ---'
python3 - <<'PY'
import hashlib
from pathlib import Path
for name in (
"skills/cuopt-install/benchmark/evals.json",
"skills/cuopt-install/evals/evals.json",
):
data = Path(name).read_bytes()
print(name)
print(" raw sha256:", hashlib.sha256(data).hexdigest())
print(" normalized-json sha256:",
hashlib.sha256(
__import__("json").dumps(
__import__("json").loads(data),
sort_keys=True,
separators=(",", ":"),
).encode()
).hexdigest())
PYRepository: NVIDIA/cuopt
Length of output: 3080
Record the Tier 3 dataset and task ID.
skills/cuopt-install/benchmark/evals.json contains 15 tasks, while skills/cuopt-install/evals/evals.json contains inst-eval-001-docker-server. The report lists one task but identifies neither input nor task ID. Add the exact input file and task ID, and reconcile the dataset digest if both inputs were evaluated.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@skills/cuopt-install/BENCHMARK.md` around lines 15 - 18, Update the benchmark
report metadata in BENCHMARK.md to identify the exact evaluated input file and
task ID, specifically distinguishing the Tier 3 dataset from the
inst-eval-001-docker-server task. Reconcile the Dataset digest and task count
with the actual inputs evaluated, recording both dataset details if both sources
were used.
Comment on lines
+74
to
+87
| | Dimension | Question | Scored signals | | ||
| |---|---|---| | ||
| | Security | Is it safe to use? | `security` (100%) | | ||
| | Correctness | Is the answer correct? | `accuracy` (100%) | | ||
| | Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) | | ||
| | Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) | | ||
| | Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` (100%) | | ||
|
|
||
| - MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/cuopt-install/SKILL.md`) | ||
| - MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/cuopt-install/SKILL.md`) | ||
| - LOW SCHEMA/unexpected_file: Unexpected 'benchmark' in skill root (`skills/cuopt-install/benchmark`) | ||
| - LOW SCHEMA/author_format: Author must be of the form 'Name <email@host>' (`skills/cuopt-install/SKILL.md`) | ||
| - Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%. | ||
| - Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL. | ||
| - Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate. | ||
| - The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold. | ||
| - Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`). | ||
| - Token efficiency is a separate report-only signal. It does not change a dimension score or the overall verdict. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Document the overall-score formula and gate input.
The report defines dimension thresholds and the verdict gate, but it does not define how Overall is derived from the five dimensions or whether the thresholds use baseline or skill-assisted scores. The displayed values imply an arithmetic mean, and both baseline efficiency values are 0%; without this rule, readers cannot reproduce the Overall values or the PASS decision. Add the aggregation formula, rounding rule, and explicit skill-assisted-score basis.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@skills/cuopt-install/BENCHMARK.md` around lines 74 - 87, Update the benchmark
scoring documentation near the dimension bands and overall verdict to define
Overall as the arithmetic mean of the five skill-assisted dimension scores,
including the rounding rule and score basis. State that the verdict gate uses
these skill-assisted scores rather than baseline values, while preserving the
existing threshold and verdict rules.
Collaborator
Author
|
/ok to test 751a554 |
Collaborator
Author
|
/ok to test 1495865 |
Collaborator
Author
|
/ok to test d22f2fd |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Trivial blank-line addition after SKILL.md frontmatter to trigger NVSkills CI — fetch updated skill card and benchmark for
cuopt-install.