Skip to content

fix(ci): docs OpenHands eval — single trigger per ready PR head #869

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rickylabs merged 3 commits into from
Jul 18, 2026
Merged

fix(ci): docs OpenHands eval — single trigger per ready PR head #869

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e0a22d3
fix(ci): make docs eval a single ready-head claim
rickylabs Jul 18, 2026
fa7d0be
chore(harness): record docs eval PR handoff
rickylabs Jul 18, 2026
a2f0e16
Merge remote-tracking branch 'origin/main' into fix/docs-eval-loop
rickylabs Jul 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
158 changes: 92 additions & 66 deletions .github/workflows/docs-openhands-eval.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,74 +1,70 @@
name: Docs OpenHands accuracy eval

# CI-level documentation-accuracy backstop. Every PR carrying `type:docs` or `area:docs` asks
# OpenHands to use Minimax M3: one of the most accurate, low-hallucination prose models, while cheap
# enough for QUICK MANUAL TESTING. This lane must exercise executable documentation claims (small
# scaffold, exact snippets, observed output), not merely reread prose. The agent-level pipeline is
# independent; see `.llm/harness/workflow/doc-audit-openhands-gate.md` for the audit contract and
# the `docs-eval:skip` on-demand escape hatch.
#
# The comment MUST use PAT_TOKEN. GitHub suppresses chained workflow events created with the default
# GITHUB_TOKEN, so a fallback would post a trigger-looking comment that never launches OpenHands.
# Dispatch exactly once for a docs PR head when the PR becomes ready for review. Maintainers may
# explicitly request the same decision path with a `/docs-eval rerun` PR comment; the durable head
# marker still prevents a second dispatch for an already-claimed SHA. `docs-eval:skip` suppresses
# dispatch, while `ci:full` makes a non-docs-labeled PR eligible for the full gate surface.

on:
pull_request:
types: [opened, synchronize, labeled]
types: [ready_for_review]
issue_comment:
types: [created]

permissions:
contents: read
issues: write
pull-requests: read

# The serialized per-PR critical section makes the marker lookup + trigger creation race-safe even
# when a ready transition and an explicit request arrive together.
concurrency:
group: docs-openhands-eval-${{ github.event.pull_request.number }}
group: docs-openhands-eval-${{ github.event.pull_request.number || github.event.issue.number }}
cancel-in-progress: false

jobs:
docs-accuracy:
name: Minimax M3 docs accuracy
if: >-
contains(github.event.pull_request.labels.*.name, 'type:docs') ||
contains(github.event.pull_request.labels.*.name, 'area:docs')
(github.event_name == 'pull_request' &&
github.event.action == 'ready_for_review' &&
(contains(github.event.pull_request.labels.*.name, 'type:docs') ||
contains(github.event.pull_request.labels.*.name, 'area:docs') ||
contains(github.event.pull_request.labels.*.name, 'ci:full'))) ||
(github.event_name == 'issue_comment' &&
github.event.action == 'created' &&
github.event.issue.pull_request &&
github.event.comment.body == '/docs-eval rerun' &&
contains(fromJson('["OWNER","MEMBER","COLLABORATOR"]'),
github.event.comment.author_association) &&
(contains(github.event.issue.labels.*.name, 'type:docs') ||
contains(github.event.issue.labels.*.name, 'area:docs') ||
contains(github.event.issue.labels.*.name, 'ci:full')))
runs-on: ubuntu-latest
timeout-minutes: 10
env:
OPENHANDS_MODEL: openrouter/minimax/minimax-m3
OPENHANDS_OUTPUT: pr-comment
OPENHANDS_ITERATIONS: '100'
SKIP_REQUESTED: ${{ contains(github.event.pull_request.labels.*.name, 'docs-eval:skip') }}
SKIP_REQUESTED: ${{ contains(github.event.pull_request.labels.*.name, 'docs-eval:skip') || contains(github.event.issue.labels.*.name, 'docs-eval:skip') }}
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
steps:
- name: Skipped on demand
if: env.SKIP_REQUESTED == 'true'
env:
REQUEST_ACTOR: ${{ github.event.sender.login }}
EVENT_ACTION: ${{ github.event.action }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
REQUEST_ACTOR: ${{ github.actor }}
PR_NUMBER: ${{ github.event.pull_request.number || github.event.issue.number }}
run: |
{
echo "## Docs OpenHands accuracy eval"
echo
echo "**Status:** skipped on demand"
echo
printf -- '- Who: `@%s` (workflow event actor)\n' "$REQUEST_ACTOR"
printf -- '- Why: PR carries the `docs-eval:skip` escape-hatch label during `%s`\n' "$EVENT_ACTION"
printf -- '- Head SHA: `%s`\n' "$HEAD_SHA"
printf -- '- Who: `@%s`\n' "$REQUEST_ACTOR"
echo '- Why: PR carries the `docs-eval:skip` escape-hatch label'
printf -- '- PR: `#%s`\n' "$PR_NUMBER"
} >> "$GITHUB_STEP_SUMMARY"

- name: Guard the open-model route
if: env.SKIP_REQUESTED != 'true'
run: |
case "$OPENHANDS_MODEL" in
*anthropic*|*claude*|*openai*|*gpt*|*gemini*|*google*)
echo "Closed-model string is prohibited for OpenHands: $OPENHANDS_MODEL" >&2
exit 1
;;
esac
if [ "$OPENHANDS_MODEL" != "openrouter/minimax/minimax-m3" ]; then
echo "Expected the owner-ratified open model openrouter/minimax/minimax-m3; got $OPENHANDS_MODEL" >&2
exit 1
fi

- name: Require a chainable comment token
if: env.SKIP_REQUESTED != 'true'
env:
Expand All @@ -80,34 +76,68 @@ jobs:
echo
echo "**Status:** failed before dispatch"
echo
echo "The repository secret \`PAT_TOKEN\` is unavailable. The workflow will not fall back to"
echo "\`GITHUB_TOKEN\`, because comments created with that token cannot trigger OpenHands."
echo "The repository secret \`PAT_TOKEN\` is unavailable. No trigger was posted."
echo "Configure a chainable PAT; \`GITHUB_TOKEN\` comments cannot launch OpenHands."
} >> "$GITHUB_STEP_SUMMARY"
exit 1
fi

- name: Post or deduplicate the OpenHands trigger
- name: Guard and claim this PR head
if: env.SKIP_REQUESTED != 'true'
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
env:
PROMPT_PATH: .llm/tools/agentic/openhands/docs-eval-prompt.md
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
with:
github-token: ${{ secrets.PAT_TOKEN }}
script: |
const owner = context.repo.owner;
const repo = context.repo.repo;
const issue_number = context.payload.pull_request.number;
const headSha = process.env.HEAD_SHA;
const promptPath = process.env.PROMPT_PATH;
const eventName = context.eventName;

if (!process.env.OPENHANDS_MODEL) {
throw new Error('OPENHANDS_MODEL is required.');
}
if (/(anthropic|claude|openai|gpt|gemini|google)/i.test(process.env.OPENHANDS_MODEL)) {
throw new Error(`Closed-model string is prohibited for OpenHands: ${process.env.OPENHANDS_MODEL}`);
}
if (process.env.OPENHANDS_MODEL !== 'openrouter/minimax/minimax-m3') {
throw new Error(
`Expected owner-ratified open model openrouter/minimax/minimax-m3; got ${process.env.OPENHANDS_MODEL}`,
);
}
let pr;
let issueNumber;
if (eventName === 'pull_request') {
pr = context.payload.pull_request;
issueNumber = pr.number;
} else {
issueNumber = context.payload.issue.number;
const response = await github.rest.pulls.get({ owner, repo, pull_number: issueNumber });
pr = response.data;
}

const labels = new Set(pr.labels.map((label) => label.name));
if (labels.has('docs-eval:skip')) {
await core.summary
.addHeading('Docs OpenHands accuracy eval')
.addRaw('**Status:** skipped on demand\n')
.addList([
`Who: @${context.actor}`,
'Why: PR carries the docs-eval:skip escape-hatch label',
`Head SHA: ${pr.head.sha}`,
])
.write();
return;
}

// Read the prompt from the trusted base SHA, not from the PR checkout. A docs PR must
// not be able to rewrite the evaluator instructions that judge that same PR.
// Read evaluator instructions from the trusted base SHA. A PR cannot rewrite the
// prompt that judges itself.
const promptResponse = await github.rest.repos.getContent({
owner,
repo,
path: promptPath,
ref: context.payload.pull_request.base.sha,
ref: pr.base.sha,
});
if (Array.isArray(promptResponse.data) || promptResponse.data.type !== 'file') {
throw new Error(`Expected a prompt file at ${promptPath}`);
Expand All @@ -117,49 +147,45 @@ jobs:
throw new Error(`${promptPath} must begin with "use harness"`);
}

const triggerLine = [
'@openhands-agent',
`model=${process.env.OPENHANDS_MODEL}`,
`output=${process.env.OPENHANDS_OUTPUT}`,
`iterations=${process.env.OPENHANDS_ITERATIONS}`,
].join(' ');
const headSha = pr.head.sha;
const marker = `<!-- docs-openhands-eval head=${headSha} -->`;
const body = `${triggerLine}\n${marker}\n\n${prompt}`;

const comments = await github.paginate(github.rest.issues.listComments, {
owner,
repo,
issue_number,
issue_number: issueNumber,
per_page: 100,
});
const identical = comments.filter((comment) => comment.body === body).at(-1);
const answered = identical && comments.some((comment) =>
comment.id > identical.id &&
String(comment.body ?? '').includes('<!-- openhands-agent-summary -->')
);

if (identical && !answered) {
const existing = comments.find((comment) => String(comment.body ?? '').includes(marker));
if (existing) {
await core.summary
.addHeading('Docs OpenHands accuracy eval')
.addRaw('Identical unanswered trigger already exists for this head SHA; not reposting.\n')
.addList([
`Head SHA: ${headSha}`,
`Existing comment: ${identical.html_url}`,
])
.addRaw('This head SHA was already claimed; no second evaluator was dispatched.\n')
.addList([`Head SHA: ${headSha}`, `Existing trigger: ${existing.html_url}`])
.write();
return;
}

const triggerLine = [
'@openhands-agent',
`model=${process.env.OPENHANDS_MODEL}`,
`output=${process.env.OPENHANDS_OUTPUT}`,
`iterations=${process.env.OPENHANDS_ITERATIONS}`,
].join(' ');
const reason = eventName === 'pull_request'
? 'ready-for-review transition'
: 'authorized /docs-eval rerun request';
const body = `${triggerLine}\n${marker}\n\n${prompt}`;
const { data: comment } = await github.rest.issues.createComment({
owner,
repo,
issue_number,
issue_number: issueNumber,
body,
});
await core.summary
.addHeading('Docs OpenHands accuracy eval')
.addRaw('Posted the guarded Minimax M3 docs-accuracy trigger.\n')
.addRaw('Claimed this head SHA and posted one guarded Minimax M3 trigger.\n')
.addList([
`Reason: ${reason}`,
`Head SHA: ${headSha}`,
`Trigger comment: ${comment.html_url}`,
`Iteration budget: ${process.env.OPENHANDS_ITERATIONS}`,
Expand Down
84 changes: 84 additions & 0 deletions .llm/runs/beta11-cli--orchestrator/slices/docs-eval-fix/research.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# Docs eval loop fix — research

Run: `beta11-cli--orchestrator`\
Slice: `docs-eval-fix`\
Lane: Codex GPT-5.6 Sol, medium (`normal_implementation`)\
Supervisor: Fable 5 orchestrator `86d308d5`

## Scope

Diagnose and repair the automated docs-accuracy workflow so eligible documentation PRs dispatch
OpenHands once when ready for review (or when explicitly re-requested), deduplicate durably per head
SHA, retain `docs-eval:skip` and `ci:full`, and surface one actionable failure without retriggering.

## Evidence

### Live failure correlation (GitHub API, 2026-07-18)

GitHub credentials were resolved in-process with `resolveGithubToken()` from
`.llm/tools/agentic/lib/agentic-lib.ts`; no token was written to disk or argv. PR metadata, workflow
runs, comments, jobs, and raw job logs were fetched through the GitHub REST API.

| PR | Durable trigger comments | OpenHands summaries | Observed behavior |
| ---- | -----------------------: | ------------------: | ----------------------------------------------------------------------------------- |
| #858 | 1 | 1 | One push-triggered eval; downstream run `29631090296` failed. |
| #861 | 14 | 14 | Fourteen head SHAs generated fourteen failed evals while the docs PR was iterating. |
| #862 | 18 | 17 | Eighteen head-SHA triggers and seventeen completed summaries at evidence time. |

Representative upstream workflow run:
[`29631086400`](https://github.com/rickylabs/netscript/actions/runs/29631086400) succeeded after
posting #858's trigger. Its downstream OpenHands run
[`29631090296`](https://github.com/rickylabs/netscript/actions/runs/29631090296) failed before any
task verdict. The same failure signature repeated on #861 and #862.

Raw downstream job logs identify the actual execution failure:

```text
ModuleNotFoundError: No module named 'fastapi'
openhands.sdk.conversation.exceptions.ConversationRunError: ... No module named 'fastapi'
```

The failing environment installed LiteLLM 1.92.0 through the current OpenHands SDK. LiteLLM imported
its MCP proxy handler on the normal completion path, which imports FastAPI; the SDK installation did
not provide it. Bootstrap succeeded, the model/provider resolved to `openrouter/minimax/minimax-m3`
/ `OPENROUTER`, and the crash happened before the model call. This rules out model credits, timeout,
and authentication as the observed cause.

### Root causes

1. The docs workflow listened to `opened`, every `synchronize`, and every `labeled` event. Draft PR
pushes therefore dispatched continuously instead of waiting for merge readiness.
2. Dedupe was intentionally revoked after any later `openhands-agent-summary`. Once a failed summary
appeared, another event on the same head was allowed to post another trigger. Across new head
SHAs, synchronize made the storm unbounded for the life of the draft.
3. The upstream check reported success after comment creation; the actionable failure appeared only
in the downstream OpenHands workflow.
4. Every downstream agent turn crashed on the missing FastAPI runtime dependency.

### #806 lineage

PR #806 introduced the trusted-base prompt, PAT-only chain, `docs-eval:skip`, head-SHA marker, and
the prior "identical unanswered trigger" rule. Its review explicitly accepted rerunning after a
summary on later label churn. That original cost tradeoff conflicts with the owner's current
exactly-once/ready-only requirement and is superseded here; the trusted prompt, open-model guard,
PAT requirement, and escape hatch remain.

### Locked correction

- Events: `pull_request: ready_for_review` and an exact authorized PR comment `/docs-eval rerun`; no
`opened`, `synchronize`, or `labeled` dispatch.
- Eligibility: `type:docs`, `area:docs`, or `ci:full`.
- Skip: `docs-eval:skip` produces an attributed successful summary and no marker/trigger.
- Dedupe: the first marker for a head SHA is permanent. A prior answer/failure never reopens the
claim. Per-PR Actions concurrency serializes marker lookup + comment creation.
- Runner: install FastAPI explicitly so LiteLLM can enter its completion path.
- Failure: token/config/runner failure produces one red run and one workflow-owned actionable
summary; no workflow event can automatically re-dispatch it.

## Stop-lines

1. No merge without green CI and opposite-family eval PASS.
2. No release publish, tag, canary, or stable action.
3. No milestone 13 closure.
4. Every sub-agent brief must repeat all stop-lines (no sub-agents are being dispatched here).
5. #824 remains drafts-only pending owner ratification.
Loading
Loading