ci(gate): wire baseline artifact + install assert-ai from git - #8
Merged
Conversation
🛡️ ASSERT safety gate
Tip No baseline was available yet. Merge a trusted run on the default branch to create one. Action version: Gate report🛡️ ASSERT — safety regression gateGate: 🆕 FirstRun 🆕 First run on this branch — no baseline to compare. The current run is |
tangym
pushed a commit
that referenced
this pull request
Aug 14, 2026
Three action features landed on main (responsibleai/assert-ai-action#6, #7, #8) between the previous commits on this branch and now. Adopt them here so the workflow uses supported inputs instead of a target-install hijack and stays useful even when a run misbehaves. - Pin the action to `@638c569`, the current tip of ASSERT assert-ai-action main. `@v1` is still anchored at v1.0.3 which predates all three features, and the paired McNemar test needs the action to be byte-identical between the baseline run and each PR run, which only a SHA guarantees. - Replace the `target-install: pip install --force-reinstall …` hack with the new first-class `assert-ai-ref: 6dd358ca7398` input (from action#6). Same practical effect (installs assert-ai from a pinned ASSERT-main SHA so the arize-phoenix<19.18.0 cap applies) but no longer conflates 'install my target' with 'install the harness'. `target-install` goes back to just the target's own requirements. - Drop the explicit `baseline-branch: main`. The event-aware default from action#8 resolves to `github.base_ref` on pull_request (correct target-branch lineage) and to `github.ref_name` on push / workflow_dispatch, so a `workflow_dispatch` on this PR branch can now upload AND find its own baseline for pre-merge validation. Re-add an explicit value if a fixed lineage is ever needed. - Add `run-timeout-minutes: 30` (from action#7). Well over the ~5-10 min expected for the 2-behavior suite; a real hang trips the cap in 30 min instead of burning the full 90-min default. Testing plan is manual: dispatch this workflow on `tangym/wire-baseline` once to seed a baseline scoped to that branch, then dispatch again to verify the paired comparison finds it and returns a non-FirstRun verdict. Merge afterwards to establish a baseline on `main`.
The workflow had neither half of the baseline contract, so every run (including the two workflow_dispatch runs and 22 scheduled runs on the parent fork) reported FirstRun. The gate never actually gated. Add the two required halves: 1. `baseline: assert-ai-baseline` + `baseline-branch: main` on the action tells find_baseline_run.py what artifact name to look up on the trusted branch. 2. A new `Publish baseline artifact` step uploads the run's own assert-ai-artifacts/ under that name on non-pull_request events. The `if: github.event_name != 'pull_request'` guard is a security requirement: a PR must not be able to overwrite the baseline it will be compared against. Also add `push: branches: [main]` so a merge to main automatically publishes a new baseline. Kept the same paths filter as the pull_request trigger so unrelated docs/scripts changes do not spend model budget on a re-baseline.
…enix crash
assert-ai==0.1.0 on PyPI ships an unbounded arize-phoenix>=15.0.0
constraint, which pip resolves to 20.1.0. That release declares a
frozen dataclass with a mappingproxy default in phoenix.trace.dsl.filter,
which Python 3.11's dataclass machinery rejects with:
ValueError: mutable default <class 'mappingproxy'>
for field boolean_names is not allowed
Every otel-extra install crashes at import time. Every current run
of this workflow hits this once real deps are installed.
ASSERT main capped arize-phoenix<19.18.0 in commit 232baefe, but no
new release has been cut to PyPI. Reinstall assert-ai from a pinned
main SHA (6dd358ca7398) via target-install so pip re-resolves and
picks up the cap.
The SHA is pinned deliberately (not 'main'), because the paired
McNemar comparison requires an identical harness build between the
baseline run and each PR run. Revert this override once assert-ai
0.1.1+ ships to PyPI.
Three action features landed on main (responsibleai/assert-ai-action#6, #7, #8) between the previous commits on this branch and now. Adopt them here so the workflow uses supported inputs instead of a target-install hijack and stays useful even when a run misbehaves. - Pin the action to `@638c569`, the current tip of ASSERT assert-ai-action main. `@v1` is still anchored at v1.0.3 which predates all three features, and the paired McNemar test needs the action to be byte-identical between the baseline run and each PR run, which only a SHA guarantees. - Replace the `target-install: pip install --force-reinstall …` hack with the new first-class `assert-ai-ref: 6dd358ca7398` input (from action#6). Same practical effect (installs assert-ai from a pinned ASSERT-main SHA so the arize-phoenix<19.18.0 cap applies) but no longer conflates 'install my target' with 'install the harness'. `target-install` goes back to just the target's own requirements. - Drop the explicit `baseline-branch: main`. The event-aware default from action#8 resolves to `github.base_ref` on pull_request (correct target-branch lineage) and to `github.ref_name` on push / workflow_dispatch, so a `workflow_dispatch` on this PR branch can now upload AND find its own baseline for pre-merge validation. Re-add an explicit value if a fixed lineage is ever needed. - Add `run-timeout-minutes: 30` (from action#7). Well over the ~5-10 min expected for the 2-behavior suite; a real hang trips the cap in 30 min instead of burning the full 90-min default. Testing plan is manual: dispatch this workflow on `tangym/wire-baseline` once to seed a baseline scoped to that branch, then dispatch again to verify the paired comparison finds it and returns a non-FirstRun verdict. Merge afterwards to establish a baseline on `main`.
Temporary reduction of test-set sizes so the workflow finishes in minutes instead of tens of minutes during pipeline shakeout. Reverts back before merge. Motivation: - Run 31788046220 hung on prompt 72/72 of the sensitivity behavior after every earlier case succeeded, using 15+ minutes of wall clock before being cancelled. Whether that's a specific tester-model request timing out silently or something else, it stalls iteration. - Even without the hang, 32 + 8 + 72 = 112 cases per run at ~15 min makes each dispatch-based validation loop slow to observe. - `min-pairs: 30` in the action means most dimensions will report `TooFewSamples` (WARN, not FAIL) at these sizes -- expected for smoke; we're validating the pipeline moves through all stages, not that the gate has statistical power. Sizes: coercion: prompt 32 -> 10, scenario 8 -> 4 (14 cases) sensitivity: prompt 72 -> 20 (20 cases) Total 34 cases. Restore to 32/8 and 72 before merging so main's baseline has the intended statistical footprint.
assert-ai-action#9 landed on main as commit f3f8fb1 and fixes the run-root resolution bug that returned the inner per-generation dir (where scores.jsonl lives) instead of the outer eval-<ts>/ dir (where test_set.jsonl lives). Without this fix, detect_test_set_drift.py rglobs from the inner dir, never finds test_set.jsonl, and downgrades every PR verdict to a misleading FirstRun with the warning "baseline is missing test_set.jsonl" -- even though the baseline artifact was correctly downloaded and both behaviors' evaluations completed cleanly. Observed on runs 31789619791, 31790967103, 31794034530, and 31794707092: all four dispatched successfully, all four found the previous run's baseline artifact, all four reported FirstRun. Bump the pin from 638c569 (main tip before #9) to f3f8fb1 (main tip after #9) so the next dispatch actually exercises the paired McNemar comparison. Move to a proper v1.x tag once one is cut.
assert-ai-action#10 fixes the cache-warming gap that was silently stuck every paired dispatch on TestSetChanged even after #9 landed. Two coupled changes in that PR: - plan_behaviors.py pins the frozen config's `suite` to the behavior slug so every dispatch of the same behavior writes into the same results/<slug>/ directory. Without this, assert-ai defaults suite_id to eval-<current-timestamp> and no two dispatches share a cache scope. - action.yml adds a warm-cache step that copies the downloaded baseline's per-slug tree into artifacts_root, so assert-ai run's activate_latest_artifacts finds latest.json and hydrates its cache from the baseline. test_set stage cache-hits (identical fingerprint) instead of regenerating from scratch, so the drift detector's SHA comparison actually matches and the paired McNemar gate can reach PASS. Pin to the tip of the PR branch (d5efa77) for immediate testing; rebump to the merge SHA once the PR lands. Also note: existing baselines on this branch were uploaded under the old results/eval-<timestamp>/ layout. The first dispatch after this bump will report FirstRun because the new suite-slug layout doesn't overlap with the old baselines. The next dispatch establishes and finds the new baseline. Only a two-dispatch onboarding window.
…tion" This reverts commit 1c6008a. The shrink was explicitly temporary -- it took total paired cases per behavior below `min-pairs: 30` and every dimension surfaced as `TooFewSamples`, so the aggregate verdict rolled up to WARN even on identical no-op runs. Fine for smoke-testing the pipeline, wrong for main. Restore before merge so main's first baseline dispatch produces the intended statistical footprint. Verified end-to-end on run 31800326805 with the shrunk sizes: paired McNemar ran, discordant pair counts were zero (identical scores across the paired comparison), verdict = WARN due to TooFewSamples. That's the expected shape for a same-code same-test-set dispatch. After this revert, the same shape is expected to surface as PASS on identical runs and as PASS/FAIL/WARN on real target-code PRs based on actual statistical evidence.
tangym
force-pushed
the
tangym/wire-baseline
branch
from
August 14, 2026 12:46
565e93f to
53562e1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two fixes to make the ASSERT safety gate actually gate:
FirstRunand the gate was toothless. Addbaseline: assert-ai-baseline+baseline-branch: mainon the action, add the pairedPublish baseline artifactupload step, and add apush: branches: [main]trigger so merges tomainnaturally publish a new baseline.assert-aifrom ASSERT'smainat a pinned SHA.assert-ai==0.1.0on PyPI ships an unboundedarize-phoenix>=15.0.0constraint that resolves to a Phoenix release that crashes at import on Python 3.11 (ValueError: mutable default <class 'mappingproxy'>). ASSERTmaincappedarize-phoenix<19.18.0in232baefe, but there is no new PyPI release yet. Reinstall from a pinned SHA (6dd358ca7398) viatarget-install:so pip re-resolves and picks up the cap.Why pin a SHA and not
mainPaired McNemar requires an identical harness build between the baseline run and each PR run. A moving branch ref would let commits drift between runs and manufacture fake regressions.
What still needs to happen after merge
main(viaworkflow_dispatchor a no-op commit on a matching path) to publish the initialassert-ai-baselineartifact.Actions → run → Artifactswith 90d retention.PASS.FAIL.Follow-up
Revert both changes once
assert-ai==0.1.1+ships to PyPI with the Phoenix cap already baked in. Marked asTODOin the workflow comments.