You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Evaluation of the `cuopt-developer` skill before publication through NVSkills-Eval.
4
+
5
+
This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use.
6
+
7
+
## Evaluation Summary
8
+
9
+
- Skill: `cuopt-developer`
10
+
- Evaluation date: 2026-06-08
11
+
- NVSkills-Eval profile: `external`
12
+
- Environment: `astra-sandbox`
13
+
- Dataset: 3 evaluation tasks
14
+
- Attempts per task: 2
15
+
- Pass threshold: 50%
16
+
- Overall verdict: PASS
17
+
18
+
## Agents Used
19
+
20
+
-`claude-code`
21
+
-`codex`
22
+
23
+
## Metrics Used
24
+
25
+
Reported benchmark dimensions:
26
+
27
+
- Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.
28
+
- Correctness: checks whether the agent follows the expected workflow and produces the correct final output.
29
+
- Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant.
30
+
- Effectiveness: checks whether the agent performs measurably better with the skill than without it.
31
+
- Efficiency: checks whether the agent uses fewer tokens and avoids redundant work.
32
+
33
+
Underlying evaluation signals used in this run:
34
+
35
+
-`security` (Security): checks for unsafe operations, secret leakage, and unauthorized access.
36
+
-`skill_execution` (Skill Execution): verifies that the agent loaded the expected skill and workflow.
-`accuracy` (Accuracy): grades final-answer correctness against the reference answer.
39
+
-`goal_accuracy` (Goal Accuracy): checks whether the overall user task completed successfully.
40
+
-`behavior_check` (Behavior Check): verifies expected behavior steps, including safety expectations.
41
+
-`token_efficiency` (Token Efficiency): compares token usage with and without the skill.
42
+
43
+
## Test Tasks
44
+
45
+
The benchmark dataset contained 3 evaluation tasks:
46
+
47
+
- Positive tasks: 3 tasks where the skill was expected to activate.
48
+
- Negative tasks: 0 tasks where no skill was expected.
49
+
- Unlabeled tasks: 0 tasks where positive/negative intent could not be inferred.
50
+
51
+
Task composition is derived from the evaluation dataset when possible. Entries with `expected_skill` set are treated as positive skill-activation cases, while entries with `expected_skill: null` are treated as negative activation cases.
52
+
53
+
## Results
54
+
55
+
| Dimension | Num |`claude-code`|`codex`|
56
+
|---|---:|---:|---:|
57
+
| Security | 6 | 100% (+0%) | 100% (+0%) |
58
+
| Correctness | 6 | 78% (-1%) | 90% (+5%) |
59
+
| Discoverability | 6 | 62% (+11%) | 66% (+7%) |
60
+
| Effectiveness | 6 | 81% (-3%) | 93% (+10%) |
61
+
| Efficiency | 6 | 61% (+15%) | 59% (+7%) |
62
+
63
+
Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available.
64
+
65
+
## Tier 1: Static Validation Summary
66
+
67
+
Tier 1 validation passed with observations. NVSkills-Eval ran 9 checks and found 9 total findings.
68
+
69
+
Top findings:
70
+
71
+
- MEDIUM QUALITY/quality_efficiency: Deeply nested references in contributing.md (`skills/cuopt-developer/SKILL.md`)
72
+
- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/cuopt-developer/SKILL.md`)
73
+
- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/cuopt-developer/SKILL.md`)
74
+
- LOW QUALITY/quality_discoverability: No '## Purpose' section (`skills/cuopt-developer/SKILL.md`)
75
+
- LOW QUALITY/quality_reliability: No prerequisites/requirements documented (`skills/cuopt-developer/SKILL.md`)
76
+
77
+
## Tier 2: Deduplication Summary
78
+
79
+
Tier 2 validation passed. NVSkills-Eval ran 2 checks and found 0 total findings.
The skill is suitable to proceed toward NVSkills-Eval publication based on this benchmark. Skill owners should keep this file with the skill and refresh it when the evaluation dataset, skill behavior, or target agents materially change.
"question": "I made two commits to fix a bug but forgot to add the DCO sign-off to both. How do I fix this before opening a PR, and what is the correct PR workflow for contributing to cuOpt as an agent?",
5
+
"expected_skill": "cuopt-developer",
6
+
"expected_script": null,
7
+
"ground_truth": "To fix missing DCO sign-off: for the most recent commit use 'git commit --amend -s'; for multiple older commits use an interactive rebase ('git rebase -i HEAD~N') and add the Signed-off-by line to each. Never use --no-verify to bypass the DCO check. For the PR workflow: contributors (including agents) must use the fork workflow — never push branches directly to NVIDIA/cuopt. Add your fork as a remote ('git remote add fork https://github.com/<username>/cuopt.git'), push the branch there, then open a PR from the fork to the upstream base branch. When an AI agent opens the PR it must be a draft PR ('gh pr create --draft') so the developer can review before reviewers are pinged. The developer marks it ready for review when satisfied.",
8
+
"expected_behavior": [
9
+
"States 'git commit --amend -s' fixes the most recent commit's missing sign-off",
10
+
"States an interactive rebase is needed to fix sign-off on multiple older commits",
11
+
"Explicitly says --no-verify must NOT be used to bypass the DCO check",
12
+
"States contributors must use the fork workflow — never push to the upstream repo directly",
13
+
"States that agent-created PRs must be draft PRs (gh pr create --draft)"
14
+
]
15
+
},
16
+
{
17
+
"id": "dev-eval-002-add-dependency-wrong-file",
18
+
"question": "I need to add a new Python test dependency to cuOpt. A colleague says I should edit conda/environments/all_cuda-132_arch-x86_64.yaml directly. Is that correct? What is the right approach?",
19
+
"expected_skill": "cuopt-developer",
20
+
"expected_script": null,
21
+
"ground_truth": "The colleague is wrong. All cuOpt dependencies are managed exclusively through the top-level dependencies.yaml — the conda/environments/*.yaml and pyproject.toml files are auto-generated and must never be edited by hand. The correct steps are: (1) Find the appropriate group in dependencies.yaml (for a Python test dependency, likely test_python_common). (2) Add the package entry under the right output_types. (3) Run 'pre-commit run --all-files' — the RAPIDS dependency-file-generator hook regenerates conda/environments/*.yaml and pyproject.toml automatically. (4) Verify the regenerated files were updated and commit them together with dependencies.yaml.",
22
+
"expected_behavior": [
23
+
"States that directly editing conda/environments/*.yaml is wrong",
24
+
"Names dependencies.yaml as the only file that should be edited by hand",
25
+
"Mentions finding the correct group (e.g., test_python_common) for a test dependency",
26
+
"States that 'pre-commit run --all-files' regenerates the downstream files via the RAPIDS hook",
27
+
"Mentions committing the regenerated files together with dependencies.yaml"
"question": "I am adding a new C++ function to cuOpt that allocates a GPU buffer and calls a CUDA kernel. A colleague wrote the allocation as 'int* d_buf = new int[N];' and error-checked the kernel with 'if (cudaGetLastError() != cudaSuccess) return;'. What is wrong with both, and what should they be replaced with?",
33
+
"expected_skill": "cuopt-developer",
34
+
"expected_script": null,
35
+
"ground_truth": "Both are wrong. Raw 'new'/'delete' for GPU memory is forbidden in cuOpt — RMM (RAPIDS Memory Manager) allocators must be used instead. The correct pattern is to use rmm::device_uvector or rmm::device_buffer (e.g., rmm::device_uvector<int> d_buf(N, stream)) which handles allocation and deallocation safely and respects CUDA stream ordering. For CUDA error checking, bare 'if (cudaGetLastError() != cudaSuccess) return;' is insufficient — cuOpt uses RAFT_CUDA_TRY which throws on error and provides a proper message: RAFT_CUDA_TRY(cudaMemcpy(...)). Runtime assertion failures should use CUOPT_EXPECTS(condition, \"message\") rather than manual if-checks. The device buffer variable name should follow the d_ prefix convention (e.g. d_buf) which is already done here, but the allocation pattern must change.",
36
+
"expected_behavior": [
37
+
"States that raw 'new'/'delete' for GPU memory is forbidden — RMM allocators must be used",
38
+
"Names rmm::device_uvector or rmm::device_buffer as the correct replacement",
39
+
"States that RAFT_CUDA_TRY is the correct macro for CUDA error checking",
40
+
"Mentions CUOPT_EXPECTS for runtime assertion-style error handling",
41
+
"Does not suggest keeping 'new int[N]' with any workaround — the replacement is mandatory"
Copy file name to clipboardExpand all lines: skills/cuopt-developer/skill-card.md
+48-8Lines changed: 48 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,12 +3,13 @@ Modify, build, test, debug, and contribute to NVIDIA cuOpt (C++/CUDA, Python, se
3
3
4
4
This skill is ready for commercial/non-commercial use. <br>
5
5
6
-
## Owner: NVIDIA <br>
6
+
## Owner
7
+
NVIDIA <br>
7
8
8
9
### License/Terms of Use: <br>
9
10
Apache 2.0 <br>
10
11
## Use Case: <br>
11
-
Developers and engineers contributing to the NVIDIA cuOpt codebase use this skill for building, testing, debugging, and submitting changes to the C++/CUDA solver core, Python bindings, REST server, and CI infrastructure. <br>
12
+
Developers and engineers who contribute to or modify the NVIDIA cuOpt codebase, covering C++/CUDA solver internals, Python bindings, server endpoints, CI pipelines, and documentation. <br>
12
13
13
14
### Deployment Geography for Use: <br>
14
15
Global <br>
@@ -18,14 +19,16 @@ Risk: Review before execution as proposals could introduce incorrect or misleadi
18
19
Mitigation: Review and scan skill before deployment. <br>
0 commit comments