[pull] main from Azure:main

.github/skills/azsdk-common-pipeline-analysis/SKILL.md

@@ -0,0 +1,50 @@
+---
+name: azsdk-common-pipeline-analysis
+license: MIT
+metadata:
+  version: "1.0.0"
+  distribution: shared
+description: 'Analyze Azure SDK CI/CD pipeline failures into a structured diagnosis, and define the required output format. Load this skill before calling azsdk_analyze_pipeline, which returns raw failure data that this skill interprets and formats. USE FOR: "pipeline failed", "build failure", "CI check failing", "tests failing in CI", "analyze pipeline", "debug SDK pipeline". DO NOT USE FOR: local build issues without pipeline context, API design review, SDK publishing, applying code fixes (instead use azsdk-common-pipeline-fixer). INVOKES: azure-sdk-mcp:azsdk_analyze_pipeline, azure-sdk-mcp:azsdk_get_pipeline_llm_artifacts, azure-sdk-mcp:azsdk_get_pr_checks, azure-sdk-mcp:azsdk_get_pipeline_status.'
+compatibility: "azure-sdk-mcp server, Azure DevOps pipeline build ID or GitHub PR link"
+---
+
+# Pipeline Analysis
+
+This skill analyzes Azure SDK CI/CD pipeline failures and provides a structured diagnosis including root cause, affected files, and concrete instructions for how to fix each issue. It does NOT apply fixes - it tells you exactly what's wrong and how to resolve it.
+
+## Rules
+
+- Load this skill **before** running `azsdk_analyze_pipeline` — the tool returns raw failure data that this skill interprets and formats per [output format](references/output-format.md).
+- Requires the `azure-sdk-mcp` server; without it, inspect logs in the Azure DevOps UI.
+- Analysis-only: never edit files or apply fixes — use `azsdk-common-pipeline-fixer` for changes.
+- Run `azsdk_analyze_pipeline` first, then categorize each failure and cite specific files/lines.
+- For infrastructure failures (network timeouts, agent crashes, throttling), recommend retry, not code changes.
+
+## MCP Tools
+
+| Tool                                             | Purpose                                        |
+| ------------------------------------------------ | ---------------------------------------------- |
+| `azure-sdk-mcp:azsdk_analyze_pipeline`           | Analyze pipeline failure (logs + test results) |
+| `azure-sdk-mcp:azsdk_get_pipeline_llm_artifacts` | Download test result artifacts from pipeline   |
+| `azure-sdk-mcp:azsdk_get_pr_checks`              | Get pipeline/check results linked to a PR      |
+| `azure-sdk-mcp:azsdk_get_pipeline_status`        | Get pipeline run status                        |
+
+## Steps
+
+1. **Identify** - Get the build ID, pipeline URL, or PR link (use the PR link when triggered from a PR comment).
+2. **Analyze** - Run `azsdk_analyze_pipeline`. It returns `failed_tasks` (log errors from failed steps) and `failed_test_titles` (failed tests grouped by file).
+3. **Categorize** each failure: test, build/compilation, validation/lint, or infrastructure.
+4. **Diagnose** - Give each failure's root cause and affected file(s)/line(s), and note if several share one root cause. See [failure patterns](references/failure-patterns.md).
+5. **Report** - Use the [output format](references/output-format.md): root cause, affected files, per-failure fix + verify command, and fixable vs infrastructure. Recommend `azsdk-common-pipeline-fixer` to apply fixes.
+
+## Examples
+
+- "My pipeline build 6447834 failed, what went wrong?"
+- PR comment: `@copilot Analyze the failed pipeline on this PR`
+- "Why is my CI red? Build ID is 6455939"
+
+## Troubleshooting
+
+- If `azsdk_analyze_pipeline` returns no data, verify the build ID is correct and the run has completed.
+- If `failed_test_titles` is empty, rely on `failed_tasks` log analysis (test-artifact upload may not be configured).
+- For a private/internal pipeline, the user may need to `az login`. See [failure patterns](references/failure-patterns.md).

.github/skills/azsdk-common-pipeline-analysis/evals/trigger.eval.yaml

@@ -0,0 +1,91 @@
+name: azsdk-common-pipeline-analysis-trigger-eval
+description: Trigger and anti-trigger tests for the azsdk-common-pipeline-analysis skill
+type: capability
+
+environment: azsdk-mcp-mock
+
+tags:
+  area: azsdk-common-pipeline-analysis
+  type: ci-gate
+
+config:
+  runs: 1
+  timeout: "90s"
+  model: claude-opus-4.6
+  executor: copilot-sdk
+
+scoring:
+  threshold: 0.8
+
+stimuli:
+  # Should trigger
+  - name: trigger-my-ci-pipeline-is-failing
+    prompt: "my Azure SDK CI pipeline build 6447834 is failing, can you help troubleshoot what went wrong?"
+    graders:
+      - type: skill-invocation
+        config:
+          required: ["azsdk-common-pipeline-analysis"]
+  - name: trigger-debug-pipeline-build-error
+    prompt: "debug the build error in my SDK generation pipeline, build 6447834"
+    graders:
+      - type: skill-invocation
+        config:
+          required: ["azsdk-common-pipeline-analysis"]
+  - name: trigger-pipeline-analysis-help
+    prompt: "I'm stuck on a failing Azure SDK pipeline — can you help me troubleshoot it?"
+    graders:
+      - type: skill-invocation
+        config:
+          required: ["azsdk-common-pipeline-analysis"]
+  - name: trigger-analyze-failing-sdk-pipeline
+    prompt: "analyze my failing Azure SDK generation pipeline https://dev.azure.com/azure-sdk/internal/_build/results?buildId=6447834 and tell me what went wrong"
+    graders:
+      - type: skill-invocation
+        config:
+          required: ["azsdk-common-pipeline-analysis"]
+  - name: trigger-ci-build-broke-after-my-change
+    prompt: "the SDK CI build broke after my change, help me troubleshoot the pipeline"
+    graders:
+      - type: skill-invocation
+        config:
+          required: ["azsdk-common-pipeline-analysis"]
+
+  # Should NOT trigger
+  # "apply the fix / auto-fix" prompts belong to azsdk-common-pipeline-fixer, not
+  # this analysis-only skill (see each skill's DO NOT USE FOR boundary).
+  - name: anti-trigger-auto-fix-and-commit
+    prompt: "auto-fix the failing tests in my pipeline and commit the changes to my PR"
+    graders:
+      - type: skill-invocation
+        config:
+          disallowed: ["azsdk-common-pipeline-analysis"]
+  - name: anti-trigger-write-a-typespec-definition
+    prompt: "write a TypeSpec definition"
+    graders:
+      - type: skill-invocation
+        config:
+          disallowed: ["azsdk-common-pipeline-analysis"]
+  - name: anti-trigger-create-a-new-skill
+    prompt: "create a new skill"
+    graders:
+      - type: skill-invocation
+        config:
+          disallowed: ["azsdk-common-pipeline-analysis"]
+  - name: anti-trigger-optimize-markdown-tokens
+    prompt: "optimize markdown tokens"
+    graders:
+      - type: skill-invocation
+        config:
+          disallowed: ["azsdk-common-pipeline-analysis"]
+  - name: anti-trigger-how-do-i-release-an-sdk-package
+    prompt: "how do I release an SDK package"
+    graders:
+      - type: skill-invocation
+        config:
+          disallowed: ["azsdk-common-pipeline-analysis"]
+  - name: anti-trigger-resolve-apiview-feedback
+    prompt: "resolve APIView feedback"
+    graders:
+      - type: skill-invocation
+        config:
+          disallowed: ["azsdk-common-pipeline-analysis"]

.github/skills/azsdk-common-pipeline-analysis/references/failure-patterns.md

@@ -0,0 +1,60 @@
+# Pipeline Failure Patterns and Resolution
+
+## Test Failures
+
+| Pattern                                  | Likely Cause                                       | Fix                                                                                                                                                            |
+| ---------------------------------------- | -------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| AssertionError (expected vs actual)      | Logic bug or unintended behavior change            | Fix the code logic or update expected value if change was intentional                                                                                          |
+| Recording file path does not exist       | Missing test recording for a new/modified test     | Best effort: use the `az` CLI to check auth (try `az account show`); if authenticated, run tests in record mode; else skip and report re-recording is required |
+| Playback failure / ResourceNotFoundError | Recorded responses don't match current API calls   | Best effort: use the `az` CLI to check auth (try `az account show`); if authenticated, re-record the session; else skip and report re-recording is required    |
+| HttpResponseError (5xx) in live tests    | Service-side issue during live test run            | Infrastructure - retry the pipeline                                                                                                                            |
+| Timeout / test hung                      | Async issue, deadlock, or slow external dependency | Add timeout, fix async logic, or mock the slow dependency                                                                                                      |
+| FileNotFoundError / missing fixture      | Test data not committed or path changed            | Add the fixture file or fix the path reference                                                                                                                 |
+
+## Build/Compilation Failures
+
+| Pattern                               | Likely Cause                                             | Fix                                                                                                                                     |
+| ------------------------------------- | -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
+| Type mismatch (mypy, pyright, tsc)    | Code passes wrong type to a function/constructor         | Fix the type at the cited file:line                                                                                                     |
+| Missing import / unresolved reference | New dependency not imported, or missing from the project | Add the import; if the package isn't a declared dependency, add it to the project manifest (e.g. pyproject.toml, .csproj, package.json) |
+| Incompatible type in generics         | Generic type parameter doesn't satisfy constraint        | Fix the type parameter or constraint                                                                                                    |
+| Syntax error                          | Malformed code                                           | Fix the syntax at the cited line                                                                                                        |
+| Missing types/models (TypeSpec)       | TypeSpec compilation issue                               | Fix TypeSpec definition, regenerate                                                                                                     |
+
+## Validation/Lint Failures
+
+| Pattern                        | Likely Cause                         | Fix                                                |
+| ------------------------------ | ------------------------------------ | -------------------------------------------------- |
+| Changelog validation error     | Missing or malformed changelog entry | Update CHANGELOG.md content                        |
+| Breaking change detected       | API surface changed incompatibly     | Revert the breaking change or add suppression      |
+| Lint/format violation          | Code doesn't match style rules       | Run the formatter (black, prettier, dotnet format) |
+| API compatibility check failed | Public API signature changed         | Update API review or revert                        |
+
+## Infrastructure Failures (Recommend Retry)
+
+| Pattern                             | Likely Cause                            | Action         |
+| ----------------------------------- | --------------------------------------- | -------------- |
+| Name or service not known (DNS)     | Transient network issue                 | Retry pipeline |
+| HTTP 429 / rate limited             | Throttling from package registry or API | Wait and retry |
+| Agent disconnected / process killed | CI agent resource exhaustion            | Retry pipeline |
+| Timeout downloading packages        | Network congestion or registry slowness | Retry pipeline |
+| Certificate / SSL error             | Transient PKI issue                     | Retry pipeline |
+
+## Cascading Failure Detection
+
+When many tests fail simultaneously, look for:
+
+- A single build/setup step that failed (all downstream tests fail with import errors)
+- A shared test fixture or setup method that throws
+- A missing environment variable or config file
+- A dependency that failed to install
+
+**Rule of thumb:** If >10 tests fail with similar error patterns, identify the common root cause rather than fixing each individually.
+
+## TypeSpec/SDK Generation Failures
+
+| Pattern                          | Likely Cause                        | Fix                                   |
+| -------------------------------- | ----------------------------------- | ------------------------------------- |
+| Missing client.tsp changes       | TypeSpec customization needed       | Use `azsdk_customized_code_update`    |
+| tspconfig.yaml emitter error     | Incorrect emitter configuration     | Fix tspconfig.yaml                    |
+| Breaking change in generated API | TypeSpec change affects SDK surface | Add `@clientName` decorator or revert |

.github/skills/azsdk-common-pipeline-analysis/references/output-format.md

@@ -0,0 +1,26 @@
+# Pipeline Analysis Output Format
+
+Write the diagnosis as markdown so it reads cleanly in a GitHub PR comment or an
+Azure DevOps pipeline summary. Order it: root cause first, then each failure,
+then a short fix summary.
+
+Include these parts:
+
+1. **Header** — build ID or PR link, package path, and overall status.
+2. **Root cause** — one or two plain sentences naming the underlying problem. If
+   failures are unrelated, list each root cause separately.
+3. **Failures** — for every failure: its category (test / build / validation /
+   infrastructure), the exact `file:line` from the analysis output, the error
+   text, and the specific fix.
+4. **Verify** — the local command that reproduces each fixable failure
+   (e.g. `python -m mypy --isolate <path>`, `dotnet test --filter <test>`).
+5. **Summary** — counts of fixable-in-code, infrastructure (retry), and
+   needs-human-input failures. Recommend `azsdk-common-pipeline-fixer` only when
+   something is code-fixable.
+
+## Rules
+
+- Always cite the real `file:line` from the analysis output; never paraphrase it.
+- Group cascading failures under their shared root cause instead of listing each.
+- Mark infrastructure failures (network, throttling, agent crash) as retry, not code.
+- See [failure patterns](failure-patterns.md) for category-to-fix mappings.

.github/skills/azsdk-common-pipeline-fixer/SKILL.md

@@ -0,0 +1,51 @@
+---
+name: azsdk-common-pipeline-fixer
+license: MIT
+metadata:
+  version: "1.0.0"
+  distribution: shared
+description: 'Automatically fix Azure SDK CI/CD pipeline failures by applying code changes and verifying locally. USE FOR: "fix pipeline failure", "fix CI", "fix failing tests", "auto-fix and commit the fix", "fix build error", "fix mypy/pylint/type-check/lint errors", "auto-fix pipeline", "resolve pipeline failure". DO NOT USE FOR: pipeline analysis (instead use azsdk-common-pipeline-analysis), API design review, SDK publishing. INVOKES: azure-sdk-mcp:azsdk_package_build_code, azure-sdk-mcp:azsdk_package_run_check, azure-sdk-mcp:azsdk_package_run_tests, azure-sdk-mcp:azsdk_verify_setup.'
+compatibility: "azure-sdk-mcp server, local azure-sdk-for-{language} clone, language build tools"
+---
+
+# Pipeline Fixer
+
+This skill automatically fixes Azure SDK CI/CD pipeline failures. It analyzes the failure, identifies the root cause, applies code changes, and verifies the fix locally using build, check, and test tools before committing.
+
+## Rules
+
+- Requires the `azure-sdk-mcp` server and a local clone of the SDK language repo.
+- Work on the PR branch — the failing code is there, not main.
+- Analyze first, then apply the minimal code change for the root cause.
+- Verify ONLY via the azsdk MCP package tools (build → check → tests); never raw shell build/test commands.
+- Never fix infrastructure failures (timeouts, crashes, throttling) — recommend retry. Iterate up to 3 times, then report.
+
+## MCP Tools
+
+| Tool                                     | Purpose                               |
+| ---------------------------------------- | ------------------------------------- |
+| `azure-sdk-mcp:azsdk_verify_setup`       | Verify local environment is ready     |
+| `azure-sdk-mcp:azsdk_package_build_code` | Build the package locally             |
+| `azure-sdk-mcp:azsdk_package_run_check`  | Run validation/lint/typecheck locally |
+| `azure-sdk-mcp:azsdk_package_run_tests`  | Run tests locally                     |
+
+**Prerequisites:** azure-sdk-mcp server required. Local clone of the affected SDK language repo. Language build tools installed.
+
+## Steps
+
+1. **Find analysis** - Reuse an existing analysis from the PR comments; if none, view the `azsdk-common-pipeline-analysis` skill.
+2. **Classify** - Fixable (test, type, lint, import, assertion errors) vs retry (infrastructure) vs escalate (breaking API, credentials).
+3. **Locate** - Identify the affected package, its path, and the files/lines to change.
+4. **Fix** - Read the failing code at the cited lines and apply the minimal fix.
+5. **Verify** - Run `azsdk_package_build_code` → `azsdk_package_run_check` → `azsdk_package_run_tests` on the package; all must pass. If the environment isn't ready, run `azsdk_verify_setup`. Never substitute raw shell build/test commands.
+6. **Iterate & commit** - On failure, revise and re-verify (max 3 attempts). Once all pass, commit with a descriptive message.
+
+## Examples
+
+- "Fix the pipeline failure for build 6447834"
+- "My Python SDK CI is failing with a type error, fix it"
+- "Auto-fix the failing tests in my PR"
+
+## Troubleshooting
+
+- If the verify tools fail because the environment isn't ready, run `azsdk_verify_setup`, report its error, and stop — don't fall back to raw shell build/test commands.