Add CI pipeline for SDK skill-efficacy evaluation

[yumnahussain]

Summary

Adds a manually-triggered GitHub Actions workflow and supporting scripts to evaluate the cosmosdb-best-practices skill set using an internal skill-efficacy evaluation harness. The pipeline measures whether the skills help an agent produce Cosmos DB SDK best-practice-compliant code, grading the generated app both behaviorally (drive its HTTP API + independently read the data store) and via static source checks.

Current scope

Scoped to a single Harbor task instance (mosaic-python) while the end-to-end pipeline is validated. The harness (shared base image, remote-backend runner, verifier, contracts) already supports additional scenarios × SDKs, which will be added in follow-up PRs.

What's included

Area	Files	Purpose
CI trigger	.github/workflows/msbench-eval.yaml	workflow_dispatch job that signs in as the triggering user via device code, installs msbench-cli, runs the eval, and opens an issue on failure
Benchmark publish	.github/workflows/publish-benchmark.yaml	Builds/publishes task images via harbor-format-curation
Orchestration	scripts/msbench-eval.py	Submits msbench-cli run --repeat N, runs a remote-backend submit preflight, polls, reads harbor eval.json, computes pass@k, checks threshold
Issue creation	scripts/create-skills-issue.py	Maps failing test classes to sdk-* rules, opens a tracking issue
Benchmark task + harness	benchmarks/cosmos-sdk-skills/**	The mosaic-python task, shared remote-backend runner, and pytest verifier that independently reads the store to confirm real persistence

Authentication (changed)

The pipeline no longer uses an Azure OIDC service principal. Instead it authenticates to Azure / the remote execution backend as the person who dispatches the run, via an interactive device-code sign-in — the same identity used for a local eval. The job prints a URL + one-time code to its logs; the dispatcher completes the browser sign-in, and all subsequent az / msbench-cli calls run as that account (which is what authorizes the backend submission and the Azure Artifacts feed read).

• New optional tenant_id workflow input (blank = az default tenant).
• A fully-unattended service-principal path is tracked separately in the service-principal setup issue; it needs extra permissions granted to the SP first.

Live Cosmos account (optional)

An optional COSMOS connection-string secret. When set, msbench-eval.py forwards it (encrypted) so the verifier grades against a real Cosmos DB account instead of the bundled emulator. Leave it unset to grade against the emulator. This enables checks that a single-node emulator can't prove (e.g. multi-region routing).

Verifier: Python client-config check

check_source.py adds TestClientConfigViaKwargs — a Python-gated static check that flags the legacy documents.ConnectionPolicy(...) construction / attribute-mutation pattern (e.g. policy.RequestTimeout = ...) and steers toward passing config as keyword arguments to CosmosClient(...). The skill rule this test cites (sdk-python-client-kwargs) lives in companion PR #225 (one-rule-per-PR); the test references the rule id as a string, so the two PRs are independent and can merge in either order.

How it works

1. Trigger via Actions → SDK Skill-Efficacy Evaluation and complete the device-code sign-in in the job logs.
2. A remote-backend submit preflight confirms the backend accepts a submission and returns a run_id before the real run.
3. Runs the task instance(s) with --repeat independent attempts (default 3).
4. Computes per-instance pass rate from the harbor eval.json.
5. If any instance falls below the threshold (default 90%), opens a GitHub issue mapping failures to specific sdk-* rules with suggested skill improvements.

Inputs

• benchmark — benchmark name (default cosmos-sdk-skills)
• repeat — independent attempts per instance (default 3)
• threshold — pass-rate threshold 0–1 (default 0.9, tentative)
• model — overrides the in-container @github/copilot CLI (blank = runner default claude-sonnet-4.6); forwarded as COPILOT_MODEL, while the eval CLI's own --model stays on a valid plugin model-map key
• tenant_id — Azure tenant to sign into (blank = az default)

Prerequisites

• The dispatcher has an Azure account entitled to submit to the remote execution backend and read access to the internal Azure Artifacts feed, and completes the device-code prompt within the timeout.
• Repo variables for the internal identifiers kept out of the public repo: MSBENCH_FEED, MSBENCH_TOKEN_RESOURCE, CES_ACR, MSBENCH_BACKEND (Settings → Secrets and variables → Actions → Variables).
• COPILOT_GITHUB_TOKEN secret — a github.com token for an account with a Copilot subscription (the default Actions token has no Copilot access); forwarded into the remote-backend container via --encrypted-env GITHUB_TOKEN.
• (Optional) COSMOS secret — a Cosmos DB connection string to grade against a live account instead of the emulator.
• If the tenant's Conditional Access blocks device-code sign-in, use a self-hosted runner where you are already az logined (or the SP path).

Companion benchmark repo

The Harbor tasks and verifier were originally authored in a separate registration repo (cosmos-sdk-skills-bench). The in-scope task + verifier are vendored into benchmarks/cosmos-sdk-skills/ here so CI can build and grade directly.

Tentative decisions (feedback welcome)

• 90% pass threshold
• Device-code (per-dispatcher) auth as the default vs. a service principal
• Vendoring the benchmark task/verifier into this repo vs. keeping it external

[Copilot]

Pull request overview

This PR adds a manually-triggered GitHub Actions pipeline plus helper scripts to run MSBench-based evaluations for Cosmos DB SDK skill guidance, generate a merged pass@k report, and (optionally) open a tracking issue when results fall below a threshold.

Changes:

• Adds a workflow_dispatch GitHub Actions workflow to authenticate via Azure OIDC, install msbench-cli, run the evaluation, and upload results.json.
• Adds scripts/msbench-eval.py to orchestrate MSBench runs, poll for completion, generate a merged report, and evaluate per-instance pass rates.
• Adds scripts/create-skills-issue.py to parse results, map failing tests to sdk-* rules, and create a GitHub issue via gh issue create.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
.github/workflows/msbench-eval.yaml	New manual CI workflow to run MSBench evaluation, optionally create an issue, and upload artifacts.
scripts/msbench-eval.py	New orchestrator script for running MSBench, merging reports, parsing pass rates, and flagging failing instances.
scripts/create-skills-issue.py	New issue-creation script to summarize failures and map them to rule files.

[Copilot]

Pull request overview

Copilot reviewed 50 out of 51 changed files in this pull request and generated 5 comments.

[NaluTripician]

Review: cosmos-sdk-skills benchmark (Python slice)

Really nice work here — the architecture is sound and the behavioral core is the strongest part. Below is a consolidated review focused on the Python (mosaic-python) slice that ships and runs today. Grouped into strengths, things to fix, and suggested next steps. Non-Python SDKs and fault-injection are noted only as future directions.

What's strong (keep it)

• The behavioral core is the right design and is hard to game. Driving the live API and then independently re-reading the emulator with the verifier's own client is exactly right — it catches apps that answer HTTP but never persist (in-memory/SQLite), proves duplicate-rejection by state rather than by scanning source for IfNoneMatch, and cross-checks the city filter against the emulator's own query result.
• Contract-driven engine cleanly separates the generic checks from scenario specifics.
• Honest labeling — check_source.py openly documents that its regex rules are the weaker half and explains why they can't be behavioral on a single node. That's the right framing; the goal should be to keep shrinking that static set.
• Isolated verifier venv + comment-stripping before regex are good anti-interference touches.

Things to fix (internal consistency — a benchmark that contradicts itself is hard to trust)

1. check_advanced_source.py never runs. It isn't in runner.sh's pytest target list and isn't imported anywhere, so ~10 rule classes (availability strategy, circuit breaker, conditional-create ETag, ETag concurrency, excluded regions, patch-counter, async-api, enum serialization, per-call request options) are dead. Either wire it in or delete it and trim the coverage docs to what actually executes.
2. Skill-name drift. The repo only has skills/cosmosdb-best-practices, but the README, runner-with-skills.sh, and create-skills-issue.py (PREFERRED_RULES_DIR = skills/cosmosdb-sdk/rules) reference a non-existent cosmosdb-sdk. Pick one name and make everything match.
3. The documented runner isn't the one the pipeline uses. README describes runner-with-skills.sh (custom --agent, AGENTS.md injected — hinted); the actual pipeline (msbench-eval.py → shared/ces/runner.sh) installs the skill for the default agent with no hint (organic). These are different experiments — pick one as canonical and document it.
4. Rule-ID drift. sdk-retry-429 (in reference/app.py and create-skills-issue.py's TEST_TO_RULE_MAP) vs sdk-retry-throttled (what check_source.py cites). Generated issues will name rules that don't exist; several map keys also point at the dead check_advanced_source.py tests.
5. Port contract is muddled (8079 / 8080 / 9080). Base image bakes APP_PORT=9080 and instruction.md says 9080, but runner.sh/conftest.py/reference/run.sh default to 8080, LIVE_MODE forces python→8080, and start-emulator.sh assumes the app is on 8080. APP_PORT_PYTHON silently doesn't apply in emulator mode. Unify to one value with one source of truth.

Test-design notes

• Binary all-or-nothing reward saturates. reward=1 requires every mandatory check to pass, so one missed static regex → 0 with no gradient — and a 0 can't distinguish "close" from "hopeless," or "skill helped" from "skill didn't." Graded/bucketed reward (already flagged as future work) is effectively required for usable signal.
• Some mandatory checks are advanced/niche. Gating pass/fail on rules the task never exercises depresses pass rates for reasons unrelated to what's being measured. Separate "must behave correctly" from "nice-to-have config."
• Only comments are stripped (not string literals), so tighten the loosest patterns to avoid a keyword-in-a-string satisfying a rule.

Suggested next steps (by impact)

• A. Make the framework internally consistent (items 1–5). Prerequisite credibility work.
• B. Add a skill-on / skill-off A/B — the core upgrade. A SKILL_ENABLED flag in ces/runner.sh (when 0, skip installing the skill), same dataset run both ways with --repeat N. This is what turns a test suite into a benchmark and measures the skill's effect rather than just pass rate.
• C. Emit per-check results and report per-rule deltas. Add --junitxml to the pytest run and aggregate per-test pass/fail across repeats and across the A/B arms. Keep the binary reward for MSBench's contract; layer analytics on top. The story becomes "the skill moved singleton-client 40%→90%, diagnostics 10%→70%," etc.
• D. Move toward a graded/bucketed reward (builds / behavioral / best-practices) so near-misses are visible and static-regex brittleness stops dominating the headline number.
• E. Add an oracle smoke test in CI — run reference/ through the verifier and assert reward=1, so a future 0 is attributable to the agent, not a harness regression.
• F. Separate infra failures from agent failures — detect "emulator never became ready" (pgcosmos warmup) and retry the instance instead of scoring it as an agent miss; otherwise flaky infra corrupts the A/B.
• G. Harden static regexes against false negatives; H. stamp provenance (model id / skill SHA / image tag / dataset) into results.json and document one-command repro; I. keep promoting static rules to behavioral wherever they're observable on the emulator.

Future directions (out of the current Python scope — roadmap)

• Fault injection to make today's static rules behavioral: a Cosmos-aware HTTP fault proxy between the app and the emulator can turn retry-on-429, transient recovery, end-to-end timeouts, and even sync-vs-async / provision-once resource behavior (CPU/RAM/threads under load) into observed behavior. Only the app should traverse the proxy; the verifier's independent read must stay unfaulted. This also produces a graded score for free.
• Multi-region behaviors (hedging, preferred/excluded regions, availability strategy, cross-region circuit breaker) can't be proven on a single-node emulator — the existing runner.sh LIVE_MODE (COSMOS connection string) path is the natural home for a real multi-region arm.
• Additional SDKs: the verifier is already language-gated. Rust (azure_data_cosmos) is the strategically interesting next target — gateway/HTTP (emulator-friendly), the skill already carries partial Rust guidance (incl. an sdk-emulator-ssl rule), and a Rust iteration already exists under testing-v2/. .NET/Java are emulator traps (Direct-mode + TLS) and should be deferred with that rationale.

Bottom line: the behavioral core is strong and worth building on. The two highest-value moves are (1) making the framework internally consistent and (2) adding a skill-on/off A/B with per-check reporting so it measures the skill's effect, not just whether agents pass.

[yumnahussain]

Addressed the things to fix and test-design notes from the review in 0fc1c23.

Things to fix

1. check_advanced_source.py never runs — deleted (it wasn't in the pytest target list nor imported anywhere; only the README referenced it). Removed from the README tree.
2. Skill-name drift cosmosdb-sdk → cosmosdb-best-practices — fixed in README.md and scripts/create-skills-issue.py (PREFERRED_RULES_DIR). The hinted runner-with-skills.sh (which used the wrong name and set COPILOT_CUSTOM_AGENT) is deleted.
3. Documented runner ≠ pipeline runner — shared/ces/runner.sh is now documented as the canonical runner. It installs the skill as a normally discoverable skill with no hint and drives the default agent, so the benchmark measures the skill's organic effect. README "CI Pipeline" section rewritten (runner + device-code auth + dropped the stale "29 rules" claim).
4. Rule-ID drift sdk-retry-throttled → sdk-retry-429 — fixed in check_source.py (all 4 language checks) and reference/README.md. Pruned the 10 TEST_TO_RULE_MAP keys that pointed at the now-deleted check_advanced_source tests.
5. Port contract — unified the app port to 9080 (the value baked into the Dockerfile and used in instruction.md/dataset.jsonl) across shared/verifier/runner.sh, conftest.py, and reference/run.sh. Corrected the stale :8080-is-the-app-port comments in start-emulator.sh (behavior unchanged — the health-server cleanup stays as hygiene).

Test-design notes

• Binary reward saturates → graded reward: agreed, but this changes the MSBench reward contract (/logs/verifier/reward.txt is binary), so I'm leaving it as deferred future work rather than folding it into this PR.
• Separate must-behave from nice-to-have: deleting check_advanced_source.py removes the niche/unexercised static gates. The remaining split stands — check_behavior.py is the core behavioral suite; check_source.py/check_skills.py are the documented weaker static half.
• Only comments are stripped, not string literals: tightened the two loosest Python regexes to require code-adjacency (retry_total\s*=|…, preferred_locations\s*=) and documented the constraint in the check_source.py docstring. I intentionally did not strip string literals wholesale — source_text also concatenates dependency manifests, and the SDK-presence checks (azure-cosmos, @azure/cosmos, Microsoft.Azure.Cosmos) legitimately match package IDs that live inside those manifest strings.

Validated: py_compile on the edited Python, bash -n on the edited shell scripts, and a repo-wide grep confirms no lingering cosmosdb-sdk / sdk-retry-throttled / check_advanced_source / runner-with-skills references.

[yumnahussain]

Scrubbed the internal MSBench identifiers out of the public repo (feed path, AAD token resource GUID, CES ACR name, CES backend). Mechanism is unchanged — the values now come from GitHub repo variables (Settings → Secrets and variables → Actions → Variables), so only placeholders appear in the YAML/scripts.

Repo variables to create before running the pipelines:

Variable	Value	Used by
MSBENCH_FEED	feed host+path, no scheme (<host>/_packaging/<feed>/pypi/simple/)	both workflows (pip index URL + trusted-host derived at runtime)
MSBENCH_TOKEN_RESOURCE	AAD resource id for the feed token	both workflows
CES_ACR	ACR name (the <name> in az acr login --name)	publish-benchmark.yaml (ACR login + injected [docker.prod])
MSBENCH_BACKEND	CES execution backend	msbench-eval.yaml → msbench-eval.py

Notes:

• These are variables, not secrets — none are credentials (the feed token is minted at runtime and masked). Repo variables aren't part of the source tree, so nothing internal is exposed publicly.
• mosaic.toml no longer commits the [docker.prod] registry; publish-benchmark.yaml appends that profile at publish time from CES_ACR. For a local prod build, add a matching [docker.prod] section yourself (documented in mosaic.toml).
• Verified: both workflows YAML-parse, msbench-eval.py compiles, both TOMLs parse, the CI-injected prod profile parses to the correct registry, and a repo-wide grep finds zero remaining internal identifiers.

[TheovanKraay]

This looks great, just a couple of things...

publish-benchmark.yaml runs on push to main under benchmarks/**, so merging this will trigger it immediately (and on every benchmarks change after). It needs CES_ACR and the Azure OIDC secrets to publish, and those aren't set up in this repo yet, so it'll fail every time, harmlessly but leaving a red run on main. Can we switch it to workflow_dispatch only for now so it just runs when we manually kick it off? Once the registry and OIDC are configured we can add the auto-trigger back. (Or leave the trigger and make the missing-CES_ACR check exit 0 and skip instead of failing, same result.).

Also msbench-eval.py still hardcodes the internal msbenchapp.azurewebsites.net URL, move that to a repo var like the others.

And a question on the eval, was leaving out the skill-on/off comparison intentional? Right now the pipeline only runs the skill-on path, so we're measuring compliance rather than efficacy and there's no delta to point at. I can see runner-no-agent.sh is already there as a baseline but not wired in, so was that planned for a follow-up, or worth adding here?

[sajeetharan]

Looks good to me overall. I agree with Theo’s comments and don’t have any additional concerns beyond those, the rest would just be nits (just added them in lines)

[yumnahussain]

Thanks Theo, good calls — all three are in the latest push.

Flipped publish-benchmark.yaml to manual-only (workflow_dispatch) so it stops red-failing on main. Left the push trigger commented out with a note so we can switch it back on once CES_ACR and OIDC are sorted.

Pulled the hardcoded URL out into a MSBENCH_RUN_ANALYSIS_URL repo var, same as MSBENCH_BACKEND — already set it on the repo. If it's ever unset the link just doesn't show up, nothing breaks.

And yeah, you're right about the skill-on/off thing — I'd assumed someone else had that experiment, but it really does belong here. So I wired runner-no-agent.sh in as a --baseline pass. It now runs both and prints a skill-off vs skill-on vs delta table plus a results-comparison.json. Put it behind a compare_baseline toggle (on by default) since it doubles the runtime, and kept the threshold gate on the skill-on run.