Skip to content

Commit 261e675

Browse files
raydocsraydocs
authored
Merge pull request #1 from raydocs/codex/hardened-benchmark-pack-v1
[codex] add hardened benchmark harness and protocol v1
2 parents 8505bf9 + 2f87f71 commit 261e675

233 files changed

Lines changed: 50849 additions & 373 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.codex/prompts/harness.evaluate.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
---
2+
description: Run Astra's bench harness, inspect failures, and summarize high-signal evaluator output.
3+
---
4+
5+
## Goal
6+
7+
You are the evaluator pass for Astra.
8+
9+
Your job is to run the bench harness, inspect the structured results, and summarize only the highest-signal failures or regressions.
10+
11+
## Process
12+
13+
1. Run `pnpm bench`
14+
2. Read `bench-results/latest.json`
15+
3. Read `bench-results/latest.feedback.md`
16+
4. Report:
17+
- failing scenarios
18+
- score deltas vs previous run
19+
- top next actions
20+
21+
## Rules
22+
23+
- Do not propose broad rewrites when a narrow fix is enough
24+
- Prefer scenario IDs, scores, and exact issue text over generic summaries
25+
- If all scenarios pass, call out imperfect passes below 100

‎.codex/prompts/harness.generate.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
---
2+
description: Implement or improve Astra code against the latest bench feedback.
3+
---
4+
5+
## Goal
6+
7+
You are the generator for Astra's evaluator harness workflow.
8+
9+
Your job is to modify code so the next `pnpm bench` run scores higher without regressing existing passing scenarios.
10+
11+
## Source Of Truth
12+
13+
Always read these first if they exist:
14+
15+
- `bench-results/latest.json`
16+
- `bench-results/latest.feedback.md`
17+
- `bench-results/latest.handoff.json`
18+
- `bench-results/latest.loop.md`
19+
- `bench-results/latest.patch-task.md`
20+
- `bench-results/latest.patch-context.md`
21+
- `bench-results/latest.patch-pass.md`
22+
- `bench-results/latest.executor.md`
23+
- `bench-results/latest.dispatch.md`
24+
- `docs/bench-harness.md`
25+
26+
## Rules
27+
28+
- Fix failing scenarios before polishing imperfect passes
29+
- Prefer the smallest defensible code change
30+
- Do not rewrite unrelated systems
31+
- Preserve existing passing behavior
32+
- Run `pnpm bench` and `pnpm test` after implementation
33+
34+
## Output
35+
36+
Return:
37+
38+
- what you changed
39+
- which scenarios improved
40+
- any remaining failing or non-100 scenarios

‎.codex/prompts/harness.loop.md‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
---
2+
description: Run a minimal generator-evaluator loop for Astra using the latest bench artifacts.
3+
---
4+
5+
## Goal
6+
7+
Run Astra's minimal evaluation-driven loop:
8+
9+
```text
10+
implement -> pnpm bench -> inspect feedback -> refine -> pnpm bench -> pnpm test
11+
```
12+
13+
## Process
14+
15+
1. Read:
16+
- `docs/bench-harness.md`
17+
- `bench-results/latest.json` if present
18+
- `bench-results/latest.feedback.md` if present
19+
- `bench-results/latest.handoff.json` if present
20+
- `bench-results/latest.generator.md` if present
21+
- `bench-results/latest.loop.json` if present
22+
- `bench-results/latest.loop.md` if present
23+
- `bench-results/latest.patch-task.md` if present
24+
- `bench-results/latest.patch-context.md` if present
25+
- `bench-results/latest.patch-pass.md` if present
26+
- `bench-results/latest.executor.md` if present
27+
- `bench-results/latest.dispatch.md` if present
28+
2. Fix regressions and failing scenarios before touching imperfect passes
29+
3. Implement the smallest change set that should improve the current highest-priority scenarios selected by the loop runner
30+
4. Use `latest.patch-task.md` as the current focused patch brief
31+
5. Use `latest.patch-context.md` as the first code context bundle before widening scope
32+
6. Use `latest.patch-pass.md` as the most compact execution brief
33+
7. Read `latest.executor.md` and respect its gate: if it says blocked, do not auto-edit blindly
34+
8. If the gate is ready and credentials are configured, run `pnpm bench:dispatch`
35+
9. Run `pnpm bench`
36+
10. Run `pnpm bench:loop`
37+
11. If failures or regressions remain, use `latest.loop.json`, `latest.loop.md`, `latest.patch-task.md`, `latest.patch-context.md`, `latest.patch-pass.md`, `latest.executor.md`, and `latest.dispatch.md` as the handoff for the next pass
38+
12. When bench is clean enough, run `pnpm test`
39+
40+
## Stop Conditions
41+
42+
- stop when all bench scenarios pass and `pnpm test` passes
43+
- or stop when the next change would require a wider architectural decision
44+
45+
## Rules
46+
47+
- Treat bench artifacts as the evaluator truth
48+
- Treat `latest.handoff.json` as the priority order and `latest.feedback.md` as the detailed explanation
49+
- Treat `latest.loop.json` / `latest.loop.md` as the current single-round execution plan
50+
- Treat `latest.patch-task.md` as the focused code-change brief for the current pass
51+
- Treat `latest.patch-context.md` as the first-pass code context bundle
52+
- Treat `latest.patch-pass.md` as the compact executor brief
53+
- Treat `latest.executor.md` as the final go/no-go gate for automatic patch attempts
54+
- Treat `latest.dispatch.md` as the external model output, not as source of truth by itself
55+
- Do not add new surfaces to the harness while fixing an existing failing run unless explicitly requested
56+
- Keep changes tightly scoped to the scored behavior

‎.github/workflows/bench-opt.yml‎

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
name: Bench Opt
2+
3+
on:
4+
# Manual trigger with configurable parameters
5+
workflow_dispatch:
6+
inputs:
7+
split:
8+
description: "Evaluation split (train, validation, holdout)"
9+
required: false
10+
default: "validation"
11+
type: choice
12+
options:
13+
- train
14+
- validation
15+
- holdout
16+
dry-run:
17+
description: "Dry-run mode (skip real execution)"
18+
required: false
19+
default: true
20+
type: boolean
21+
allow-promotion:
22+
description: "Allow promotion if gates pass"
23+
required: false
24+
default: false
25+
type: boolean
26+
27+
# Scheduled: daily at 04:00 UTC, weekly full run on Sundays
28+
schedule:
29+
- cron: "0 4 * * *" # daily
30+
- cron: "0 6 * * 0" # weekly (Sunday)
31+
32+
# PR-triggered when labeled with bench-opt
33+
pull_request:
34+
types: [labeled]
35+
36+
permissions:
37+
contents: read
38+
39+
concurrency:
40+
group: bench-opt-${{ github.ref }}
41+
cancel-in-progress: true
42+
43+
jobs:
44+
bench-opt:
45+
# Only run on manual/schedule, or when the bench-opt label is applied
46+
if: >-
47+
github.event_name == 'workflow_dispatch' ||
48+
github.event_name == 'schedule' ||
49+
(github.event_name == 'pull_request' && github.event.label.name == 'bench-opt')
50+
runs-on: ubuntu-latest
51+
timeout-minutes: 30
52+
53+
steps:
54+
- name: Checkout repository
55+
uses: actions/checkout@v4
56+
57+
- name: Setup pnpm
58+
uses: pnpm/action-setup@v4
59+
with:
60+
version: 10
61+
62+
- name: Setup Node.js
63+
uses: actions/setup-node@v4
64+
with:
65+
node-version: 22
66+
cache: pnpm
67+
68+
- name: Install dependencies
69+
run: pnpm install --frozen-lockfile
70+
71+
- name: Type check
72+
run: pnpm type-check
73+
74+
- name: Run bench-opt tests
75+
run: pnpm test -- --reporter=verbose bench-opt/
76+
77+
- name: Run bench-opt optimizer
78+
run: pnpm bench:opt
79+
env:
80+
BENCH_OPT_SPLIT: ${{ github.event.inputs.split || 'validation' }}
81+
BENCH_OPT_DRY_RUN: ${{ github.event.inputs.dry-run || 'true' }}
82+
BENCH_OPT_ALLOW_PROMOTION: ${{ github.event.inputs.allow-promotion || 'false' }}
83+
84+
- name: Install Playwright browsers
85+
run: npx playwright install chromium --with-deps
86+
87+
- name: Run live smoke scenarios
88+
run: |
89+
set -euo pipefail
90+
echo "Running live smoke scenarios..."
91+
pnpm bench:live -- --scenario bench-live/page-translation-article-basic-source-bilingual
92+
pnpm bench:live -- --scenario bench-live/interaction-priority-basic
93+
pnpm bench:live -- --scenario bench-live/input-translation-basic
94+
pnpm bench:live -- --scenario bench-live/frame-coordination-basic
95+
echo "Live smoke scenarios complete."
96+
env:
97+
PLAYWRIGHT_BROWSERS_PATH: 0
98+
99+
- name: Upload live results
100+
if: always()
101+
uses: actions/upload-artifact@v4
102+
with:
103+
name: bench-live-results-${{ github.run_id }}
104+
path: bench-live-results/
105+
retention-days: 30
106+
if-no-files-found: ignore
107+
108+
- name: Upload bench-opt results
109+
if: always()
110+
uses: actions/upload-artifact@v4
111+
with:
112+
name: bench-opt-results-${{ github.run_id }}
113+
path: |
114+
bench-opt-results/
115+
retention-days: 30
116+
if-no-files-found: ignore
117+
118+
- name: Upload promotion artifacts
119+
if: always()
120+
uses: actions/upload-artifact@v4
121+
with:
122+
name: bench-opt-promotions-${{ github.run_id }}
123+
path: |
124+
bench-opt-results/promotions/
125+
bench-opt-results/publish/
126+
bench-opt-results/rollbacks/
127+
retention-days: 90
128+
if-no-files-found: ignore
129+
130+
- name: Upload store artifacts
131+
if: always()
132+
uses: actions/upload-artifact@v4
133+
with:
134+
name: bench-opt-store-${{ github.run_id }}
135+
path: |
136+
bench-opt-results/store/
137+
retention-days: 90
138+
if-no-files-found: ignore
139+
140+
- name: Summary
141+
if: always()
142+
run: |
143+
echo "## Bench Opt Results" >> $GITHUB_STEP_SUMMARY
144+
echo "" >> $GITHUB_STEP_SUMMARY
145+
echo "- **Run ID:** ${{ github.run_id }}" >> $GITHUB_STEP_SUMMARY
146+
echo "- **Trigger:** ${{ github.event_name }}" >> $GITHUB_STEP_SUMMARY
147+
echo "- **Split:** ${{ github.event.inputs.split || 'validation' }}" >> $GITHUB_STEP_SUMMARY
148+
echo "- **Dry Run:** ${{ github.event.inputs.dry-run || 'true' }}" >> $GITHUB_STEP_SUMMARY
149+
echo "" >> $GITHUB_STEP_SUMMARY
150+
if [ -f bench-opt-results/latest.json ]; then
151+
echo "### Latest Report" >> $GITHUB_STEP_SUMMARY
152+
echo '```json' >> $GITHUB_STEP_SUMMARY
153+
head -50 bench-opt-results/latest.json >> $GITHUB_STEP_SUMMARY
154+
echo '```' >> $GITHUB_STEP_SUMMARY
155+
fi
156+
if [ -f bench-live-results/latest.result.json ]; then
157+
echo "" >> $GITHUB_STEP_SUMMARY
158+
echo "### Live Smoke Results" >> $GITHUB_STEP_SUMMARY
159+
echo '```json' >> $GITHUB_STEP_SUMMARY
160+
head -30 bench-live-results/latest.result.json >> $GITHUB_STEP_SUMMARY
161+
echo '```' >> $GITHUB_STEP_SUMMARY
162+
fi
163+
if [ -d bench-opt-results/promotions ]; then
164+
echo "" >> $GITHUB_STEP_SUMMARY
165+
echo "### Promotion Artifacts" >> $GITHUB_STEP_SUMMARY
166+
ls -la bench-opt-results/promotions/ 2>/dev/null >> $GITHUB_STEP_SUMMARY || echo "No promotion artifacts." >> $GITHUB_STEP_SUMMARY
167+
fi

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,10 @@ dist/
99
.env.production
1010
logs/
1111
bench-results/
12+
bench-opt-results/
13+
bench-live-results/
14+
.bench-opt/
15+
.claude/
1216
DerivedData/
1317
ios/build/
1418
*.xcuserstate

‎agent-config/README.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Agent Configuration
2+
3+
This directory holds the declarative configuration files that the bench-opt
4+
optimizer uses to drive agent behavior. The optimizer reads these files at
5+
the start of each trial, applies candidate mutations, and writes the
6+
mutated variants into the trial worktree so the bench harness can evaluate
7+
them.
8+
9+
## Structure
10+
11+
```
12+
agent-config/
13+
defaults.json - Default agent configuration (model routing,
14+
temperature, system prompts, retry policy)
15+
tool-registry.json - Registry of available tools with metadata
16+
graph-templates/ - Execution graph templates that describe how
17+
tools, prompts, and evaluators are wired together
18+
translation-flow.json - Example graph for the translation pipeline
19+
```
20+
21+
## How it works
22+
23+
1. **defaults.json** contains the baseline agent settings. The optimizer
24+
never mutates this file directly; instead it generates a *candidate*
25+
JSON that overrides specific keys.
26+
27+
2. **tool-registry.json** enumerates every tool the agent can invoke.
28+
Each entry has an `enabled` flag, parameter metadata, and a
29+
`critical` marker that prevents the optimizer from disabling
30+
safety-critical tools.
31+
32+
3. **graph-templates/** hold execution graphs expressed as JSON. A graph
33+
is a DAG of typed nodes (tool, prompt, evaluator, router) connected
34+
by edges with optional conditions. The optimizer can add, remove, or
35+
rewire nodes to explore alternative execution strategies.
36+
37+
## Consumed by
38+
39+
- `bench-opt/mutate-tools.ts` reads and mutates `tool-registry.json`
40+
- `bench-opt/mutate-graph.ts` reads and mutates graph templates
41+
- `bench-opt/candidates/tool-config.ts` generates tool mutation candidates
42+
- `bench-opt/candidates/agent-graph.ts` generates graph mutation candidates

‎agent-config/defaults.json‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
{
2+
"schemaVersion": 1,
3+
"modelRouting": {
4+
"primary": {
5+
"provider": "openai",
6+
"model": "gpt-5.4-nano",
7+
"temperature": 0.3,
8+
"maxTokens": 4096
9+
},
10+
"fallback": {
11+
"provider": "gemini",
12+
"model": "gemini-3.1-flash-lite-preview",
13+
"temperature": 0.2,
14+
"maxTokens": 4096
15+
},
16+
"routingStrategy": "primary-with-fallback"
17+
},
18+
"systemPrompts": {
19+
"translation": "You are a professional translator. Translate the following text accurately, preserving formatting and meaning.",
20+
"summarization": "You are a concise summarizer. Extract the key points from the provided content.",
21+
"extraction": "You are an information extractor. Identify and return the requested structured data from the content."
22+
},
23+
"retryPolicy": {
24+
"maxRetries": 2,
25+
"backoffMs": 500,
26+
"retryableErrors": ["rate_limit", "timeout", "server_error"]
27+
},
28+
"timeouts": {
29+
"requestMs": 30000,
30+
"totalMs": 120000
31+
},
32+
"concurrency": {
33+
"maxParallelRequests": 3,
34+
"queueStrategy": "fifo"
35+
}
36+
}

0 commit comments

Comments
 (0)