Skip to content

Fix tutorial evaluations: use default model and correct Risky Containers docs path - #119

Open
ruokun-niu wants to merge 2 commits into
mainfrom
tutorial-evaluation-update
Open

ruokun-niu wants to merge 2 commits into
mainfrom
tutorial-evaluation-update

Conversation

@ruokun-niu

@ruokun-niu ruokun-niu commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #116.

Resolves two evaluation failures:

  • The pinned claude-opus-4.7 model is no longer available.
  • The Risky Containers documentation path points to an old directory containing screenshots but no tutorial entry point.

Changes

  • Remove the pinned model from all three scheduled evaluation attempts.
  • Use Copilot CLI’s default model for manual evaluations unless an explicit model override is supplied.
  • Correct the Risky Containers documentation path in both workflows.
  • Document default model behavior and manual overrides.

Validation

All five tutorial evaluations passed in this manual workflow run on the updated PR branch.

The automatic PR checks still fail because pull_request_target uses the base branch’s workflow, which retains the outdated model flag. The manual run uses the updated workflow and confirms the changes work.

Type of change

Bug fix for tutorial evaluation workflows.

Signed-off-by: Ruokun Niu <ruokunniu@microsoft.com>
Copilot AI balanced review requested due to automatic review settings October 14, 2026 16:59
@ruokun-niu ruokun-niu self-assigned this Oct 14, 2026
Comment thread .github/workflows/tutorial-evaluation-scheduled.yml Dismissed
Comment thread .github/workflows/tutorial-evaluation-scheduled.yml Dismissed
Comment thread .github/workflows/tutorial-evaluation-scheduled.yml Dismissed

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The model-selection changes are sound; remaining feedback concerns only the PR’s issue-closing reference.

1 open finding
What changed in this PR

Updates tutorial evaluations to use Copilot CLI’s default model while preserving manual overrides.

Changes:

  • Removes the pinned model from all three scheduled attempts.
  • Makes the manual model override optional.
  • Documents default and override behavior.
File Description
README.md Documents model selection behavior.
.github/​workflows/​tutorial-evaluation.yml Conditionally passes an explicit model override.
.github/​workflows/​tutorial-evaluation-scheduled.yml Uses the CLI default for all attempts.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/workflows/tutorial-evaluation-scheduled.yml
@ruokun-niu ruokun-niu changed the title switch to default model Use Copilot CLI’s default model for tutorial evaluations Oct 14, 2026
@ruokun-niu
ruokun-niu deployed to tutorial-evaluation October 14, 2026 18:00 — with GitHub Actions Active
@ruokun-niu
ruokun-niu deployed to tutorial-evaluation October 14, 2026 18:00 — with GitHub Actions Active
@ruokun-niu
ruokun-niu deployed to tutorial-evaluation October 14, 2026 18:00 — with GitHub Actions Active
Signed-off-by: Ruokun Niu <ruokunniu@microsoft.com>
@ruokun-niu
ruokun-niu deployed to tutorial-evaluation October 14, 2026 18:46 — with GitHub Actions Active
@ruokun-niu
ruokun-niu deployed to tutorial-evaluation October 14, 2026 18:46 — with GitHub Actions Active
@ruokun-niu
ruokun-niu deployed to tutorial-evaluation October 14, 2026 18:46 — with GitHub Actions Active
@ruokun-niu
ruokun-niu deployed to tutorial-evaluation October 14, 2026 18:46 — with GitHub Actions Active
@ruokun-niu
ruokun-niu deployed to tutorial-evaluation October 14, 2026 18:46 — with GitHub Actions Active
@ruokun-niu

Copy link
Copy Markdown
Contributor Author

All five tutorial evaluations passed in this manual workflow run using the updated PR branch. The automatic PR checks still fail because pull_request_target uses the base branch’s workflow, which retains the outdated model flag, rather than the updated workflow in this PR.

@ruokun-niu ruokun-niu changed the title Use Copilot CLI’s default model for tutorial evaluations Fix tutorial evaluations: use default model and correct Risky Containers docs path Oct 14, 2026
@ruokun-niu
ruokun-niu marked this pull request as ready for review October 14, 2026 20:07
@ruokun-niu
ruokun-niu requested a review from a team as a code owner October 14, 2026 20:07

This branch was successfully deployed

1 active deployment
tutorial-evaluation — 90956836 Deployed Oct 14, 2026 by ruokun-niu via evaluate (absence-of-change) #95
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scheduled tutorial evaluation failed - 2026-09-18

4 participants