feat(skills): runnable determinism gate + fix broken gate providers + plugin leak - #62
Merged
Merged
Conversation
…rop eval-run leak Make the loop's pre-edit regression gate actually runnable, and fix what did not work: - 'make determinism [SKILL=<name>]': run the promptfoo determinism config for skills that define one. Needs API keys + Langfuse, so NOT wired into make test / CI. - FIX broken gate providers: finish-feature/resolve-repo/new-feature/review used 'exec: bash eval/lib/run-skill.sh', which errored with 'bash: eval/lib/run-skill.sh: No such file or directory' because promptfoo runs the exec provider from the CONFIG's own directory (skills/<name>/), and promptfoo only accepts a file that resolves LOCALLY to that dir (a '../../' path is rejected as 'no valid files'). Give each skill a local eval-run.sh wrapper (matching trace-review's proven pattern) that execs the shared ../../eval/lib/run-skill.sh via an absolute path. Verified the chain reaches claude. - FIX plugin leak: exclude eval-run.sh from the shipped plugin (dev-only exec provider, references bin/devflow / run-skill.sh via repo-relative paths absent in a plugin install) — same class as determinism.promptfooconfig.yaml. - CLAUDE.md: document the gate + the config-dir-local provider rule. Note: the weekly-cadence gap is already covered by the merged trace-review skill's 'devflow trace-review schedule --backend cron|claude' subcommand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Fixes the determinism regression gate workflow for multiple skills by making promptfoo exec: providers runnable from the config directory, and ensures dev-only harness files don’t ship in the distributed plugin.
Changes:
- Add per-skill local
eval-run.shwrappers and update determinism promptfoo configs to useexec: bash eval-run.sh. - Exclude
eval-run.sh(and existing determinism configs) from plugin output to avoid shipping broken dev-only providers. - Add
make determinism [SKILL=<name>]entry point and document the provider path rule inCLAUDE.md.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| skills/review/eval-run.sh | Adds local promptfoo exec-provider wrapper to call shared runner via absolute path. |
| skills/review/determinism.promptfooconfig.yaml | Switches provider to exec: bash eval-run.sh to match promptfoo’s config-dir execution. |
| skills/resolve-repo/eval-run.sh | Adds local promptfoo exec-provider wrapper. |
| skills/resolve-repo/determinism.promptfooconfig.yaml | Switches provider to exec: bash eval-run.sh. |
| skills/new-feature/eval-run.sh | Adds local promptfoo exec-provider wrapper. |
| skills/new-feature/determinism.promptfooconfig.yaml | Switches provider to exec: bash eval-run.sh. |
| skills/finish-feature/eval-run.sh | Adds local promptfoo exec-provider wrapper. |
| skills/finish-feature/determinism.promptfooconfig.yaml | Switches provider to exec: bash eval-run.sh. |
| scripts/build-skills.sh | Treats eval-run.sh as dev-only so it is excluded from shipped plugin output. |
| Makefile | Adds determinism target to run promptfoo gates across skills (or one skill). |
| devflow-plugin/skills/trace-review/eval-run.sh | Removes leaked dev-only exec-provider from the shipped plugin tree. |
| CLAUDE.md | Documents how to run determinism gate + promptfoo provider path constraints. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| determinism: ## Run the promptfoo determinism gate for skills with a config (needs API keys + Langfuse; not in CI). One skill: make determinism SKILL=<name> | ||
| @command -v npx >/dev/null 2>&1 || { echo "determinism: npx (Node) is required"; exit 1; } | ||
| @[ -f "$(HOME)/.config/zsh/secrets" ] && . "$(HOME)/.config/zsh/secrets" 2>/dev/null || true; \ |
Comment on lines
+115
to
+117
| runs each config's `exec:` provider from that config's own directory, so provider paths are | ||
| config-dir-relative (`../../eval/lib/run-skill.sh` for the claude-spawning skills, a local | ||
| `eval-run.sh` for trace-review). This is the loop's pre-edit regression gate. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
bash: eval/lib/run-skill.sh: No such file. Fixed.eval-run.sh(a dev-only promptfoo exec provider) was shipping in the plugin with a repo-relative path that does not exist in a plugin install. Excluded.make determinism [SKILL=<name>]— one entry point to run the gate.Context
Follows #58 (single-source generation) and #57 (trace-review). Closes loop-glue gaps from the trace-improvement spike. Branched off current main (v0.13.0).
Findings (verify-first)
exec:provider from the config file's own directory (proven: trace-review's localeval-run.shpasses 6/6 from repo root) and only accepts a provider file that resolves locally to that dir — a../../path is rejected ("no valid files"). The 4 non-trace-review configs usedeval/lib/run-skill.sh, which resolved to a nonexistentskills/<name>/eval/lib/run-skill.shand errored, so their gate never ran.eval-run.shwrapper (matching trace-review's proven pattern) that execs the shared../../eval/lib/run-skill.shvia an absolute path. Verified the chain reaches claude.Changes
skills/{finish-feature,resolve-repo,new-feature,review}/eval-run.shdeterminism.promptfooconfig.yamlexec: bash eval-run.shscripts/build-skills.sheval-run.shfrom the plugin (dev-only)Makefilemake determinism [SKILL=<name>](needs API keys + Langfuse; not in CI)CLAUDE.mdVerification
make test(skills-check + flows-check + smoke): PASS.make skills-guard: OK.make determinism SKILL=trace-review: 6/6 passed.Note
The weekly-cadence gap is already covered by the merged trace-review skill's
devflow trace-review schedule --backend cron|claude. Next up (separate PR): the spike's components (b) SKILL.md -> Langfuse prompt mirror, (c) trace-derived datasets, (d) score-drop alerts.