Repository navigation
Conversation
Runs each skill's eval prompt through claude/codex/opencode/pi headlessly in isolated working dirs (skill + deps symlinked into .claude/skills and .agents/skills), captures each agent's session transcript, and uses claude as an evidence-grounded judge scoring per-assertion + whether the skill actually loaded. - gen-tasks.mjs: build tasks.json from evals + manifest (deps, agent compat) - orchestrate.mjs: per-agent runners, process-group kill, --rejudge mode - judge/: schema + instructions, JSON enforced via prompt - claude-team auto-detect with claude fallback; pi via testbench/.env
opencode resolved its workspace and skill discovery from $PWD (inherited repo root), writing outputs to the repo root and missing the run-dir skill. Set PWD=cwd and git init each run cwd so every agent stays contained and discovers the symlinked skill. Also aligns opencode model to gpt-5.5.
- eval-overrides.json: fix unrunnable steel-skill-creator URL and the over-strict steel-browser field-name assertion without touching skills/ - usage.mjs: extract tokens + USD cost per agent (claude/opencode/pi cost, codex tokens-only) - report.mjs: per-agent cost/token rollup, per-run cost matrix - gen-tasks/orchestrate: apply overrides, persist usage in meta
Query the Steel API for sessions created during each run's time window and record count + total duration/credits/proxy bytes in meta.json. Surface the totals in the per-agent rollup and per-run cost matrix in REPORT.md.
… note - steel-sessions.mjs: query Steel API for sessions opened during a run's time window; record count + duration/credits/proxy bytes in meta - orchestrate: capture started/ended, attach steel_sessions to meta + results - report: steel sessions + session-time columns in rollup and per-run matrix - README: warn to clean global browser/Steel skills before runs; document session tracking and its sequential-attribution caveat
Symlinked skills resolved to the real repo skill dir, so a skill whose scripts write cwd-relative output (e.g. steel-session-debugging's .steel-debug/) escaped the git-init+PWD jail and polluted the repo when an agent ran them from the skill dir. Copy the skill tree into cwd/ so those writes stay contained. Skills are tiny (<128K), so copy cost is negligible.
…vals
Replaces the fake sess_123 placeholder with a real, deterministically
failed Steel session shared across the matrix:
- seed/seed-session.mjs drives a rejected login on the-internet.herokuapp.com
(quotes.toscrape.com accepts any creds, so it can't fail), producing a
session with real logs/traces, and writes runs/_fixtures.json.
- orchestrate seeds once before the matrix and substitutes {{session_id}}
into prompts/assertions at run time; the resolved prompt is persisted to
meta.json so --rejudge scores against what actually ran. --no-seed reuses
an existing fixture.
- gen-tasks flags tasks that consume the fixture (prompts referencing the
token); eval-overrides point steel-session-debugging and steel-reliability
at the real session. Skills themselves are never modified.
steel-session-debugging now reads genuine session diagnostics instead of a
hypothetical id.
…eal Steel For skills that produce runnable code (EXEC_SKILLS), the harness now runs the agent's output instead of only judging the source: - orchestrate finds the generated entry (imports steel-sdk / builds a CDP connection), runs it with tsx against a real Steel session (STEEL_API_KEY injected), and records dev_exec (exit code, whether it printed the page title) into meta.json + exec-stdout.txt. The judge scores an assertion on the execution result. The Steel-session window now closes after exec so the script's own session is counted. - package.json pins steel-sdk + playwright + tsx; deps resolve from testbench/node_modules via upward lookup, so a generated script in any run cwd runs unmodified. playwright installs with no browser download. - eval-overrides adds the execution assertion for steel-developer; README documents the seed chain, EXEC_SKILLS, and the npm install prerequisite. Verified end-to-end: claude's script runs to exit 0, prints "Example Domain", opens 1 real session, judge passes.
Some agents (pi) emit NUL bytes in their transcript; passed as a CLI arg to `claude -p` they throw "argument must be a string without null bytes" and the judge never runs (seen on pi :: steel-session-debugging). Strip NULs from the assembled judge prompt before spawning.
Emit Codex-native plugin artifacts from manifest.json: a marketplace catalog at .agents/plugins/marketplace.json and a .codex-plugin/plugin.json per skill, keeping manifest.json the single source of truth. scripts/generate-codex.mjs writes the files, or verifies them with --check; wire --check into CI alongside the marketplace check. Each plugin bundles its skill via `skills: "./"` (SKILL.md at the skill-dir root). Hooks are omitted (Codex plugin hooks are documented but not currently loaded). Verified against codex 0.137.0: marketplace add, plugin list, and install of steel-browser all succeed and the skill resolves from the install cache.
nibzard
force-pushed
the
feat/codex-plugins
branch
from
June 4, 2026 18:07
f2c02c7 to
41f0e24
Compare
Contributor
Author
|
Closing to reopen clean; this PR incorrectly bundled testbench changes. Will open a new PR with only the Codex plugin commit. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Emit Codex-native plugin artifacts from manifest.json: a marketplace catalog at .agents/plugins/marketplace.json and a .codex-plugin/plugin.json per skill, keeping manifest.json the single source of truth.
Verified against codex 0.137.0: marketplace add, plugin list, and install of steel-browser all succeed and the skill resolves from the install cache.