Sync .github/skills directory with azure-sdk-tools for PR 16380#10150
Open
azure-sdk-automation[bot] wants to merge 5 commits into
Open
Sync .github/skills directory with azure-sdk-tools for PR 16380#10150azure-sdk-automation[bot] wants to merge 5 commits into
azure-sdk-automation[bot] wants to merge 5 commits into
Conversation
Wires environment.skills: [<skill dirs>] into the mock workflow eval YAML specs so Vally loads the relevant skill per-eval instead of relying on --skill-dir at the pipeline level.
- Add missing environment.skills block to sdk-local-update-metadata-001 (all other positive stimuli in this file already had it) - Replace ambiguous/fictional 'contoso' package paths with the real fixture package (sdk/compute/azure-mgmt-compute, azure-resourcemanager-compute) so the model doesn't need to ask for clarification on a made-up path - Add explicit version/release-type/date to the version-update stimuli since the tool genuinely requires that info Verified locally: suite score went from ~42% to consistently 83-92% across repeated runs (threshold 80%).
The trigger.eval.yaml suite reproducibly failed at ~70%% because vague/ informational prompts (SDK release checklist, walk me through the release process, how to publish a new SDK version, release a new version of azure-sdk) weren't covered by the skill's narrow trigger phrase list, so the model didn't reliably invoke the skill. Added those phrasings to the description, Triggers section, and Examples, and clarified that the skill should also handle how-to/ checklist style questions, not just direct action requests. Verified locally: trigger.eval.yaml went from 70%% to 100%% across two repeated runs (threshold 80%%).
Per Copilot PR review feedback: anti-trigger and negative stimuli in these eval specs didn't set environment.skills, so the skill was never loaded/available for those prompts. That made the disallowed skill-invocation / tool-calls grader checks trivially true (the skill literally couldn't be invoked), providing no real signal. Added environment.skills: [..] to all anti-trigger/negative stimuli in: - azsdk-common-apiview-feedback-resolution/evals/trigger.eval.yaml - azsdk-common-pipeline-fixer/evals/trigger.eval.yaml - azsdk-common-pipeline-analysis/evals/trigger.eval.yaml - azsdk-common-prepare-release-plan/evals/trigger.eval.yaml - azsdk-common-sdk-release/evals/trigger.eval.yaml - azsdk-common-generate-sdk-locally/evals/eval.yaml (sdk-local-negative-001) Verified locally: all 6 suites still pass with the skill loaded and available - the model correctly declines to invoke it on out-of-scope prompts. No new failures introduced (5 trigger suites at 100%, generate-sdk-locally-eval at 91.7%, same pre-existing unrelated failure as before on sdk-local-analyzer-errors-001).
azure-sdk-automation
Bot
added
EngSys
|
Azure Pipelines: 1 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot could not run the full agentic suite for this review because it was automatically requested on a bot-authored pull request. Request a review from Copilot under Reviewers to retry with the full agentic suite. Improved support for bot-authored pull requests is coming soon.
Updates Azure SDK “skills” definitions and evaluation stimuli to improve routing/triggering for release workflows (release plans, generation, pipeline analysis/fixing, and APIView resolution) and to add a new pipeline-based SDK generation skill.
Changes:
- Added per-stimulus
environment.skillsloading to multiple eval YAMLs to ensure the intended skill is available during evaluation. - Expanded trigger language and examples in release/release-plan skill docs to include more “how-to” / process-oriented prompts.
- Added a new
azsdk-common-generate-sdk-pipelineskill and updated the local-generation skill docs/evals to steer users to the pipeline flow when appropriate.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| .github/skills/azsdk-common-sdk-release/evals/trigger.eval.yaml | Loads skill context during trigger/anti-trigger eval stimuli. |
| .github/skills/azsdk-common-sdk-release/SKILL.md | Expands trigger wording/examples to cover release checklists/process questions. |
| .github/skills/azsdk-common-prepare-release-plan/evals/trigger.eval.yaml | Loads skill context during trigger/anti-trigger eval stimuli. |
| .github/skills/azsdk-common-prepare-release-plan/SKILL.md | Clarifies/expands trigger wording for release plan requests. |
| .github/skills/azsdk-common-pipeline-fixer/evals/trigger.eval.yaml | Loads skill context during trigger/anti-trigger eval stimuli. |
| .github/skills/azsdk-common-pipeline-analysis/evals/trigger.eval.yaml | Loads skill context during trigger/anti-trigger eval stimuli. |
| .github/skills/azsdk-common-generate-sdk-pipeline/SKILL.md | Introduces a new skill for pipeline-based SDK generation per release plan/language. |
| .github/skills/azsdk-common-generate-sdk-locally/evals/eval.yaml | Enriches prompts and loads skill context during eval stimuli. |
| .github/skills/azsdk-common-generate-sdk-locally/SKILL.md | Adds guidance to route “pipeline/all languages/no local clone” requests away from local generation. |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/trigger.eval.yaml | Loads skill context during trigger/anti-trigger eval stimuli. |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/eval.yaml | Loads skill context during tool-call eval stimulus. |
helen229
approved these changes
Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sync .github/skills directory with azure-sdk-tools for PR Azure/azure-sdk-tools#16380 See eng/common workflow