diff --git a/.claude/skills/running-tend/SKILL.md b/.claude/skills/running-tend/SKILL.md index 34f97f43..46257fef 100644 --- a/.claude/skills/running-tend/SKILL.md +++ b/.claude/skills/running-tend/SKILL.md @@ -25,7 +25,7 @@ Tend has Claude-powered workflows beyond the generated `tend-*` set: | Workflow | File | Schedule | Purpose | |----------|------|----------|---------| -| `review-reviewers` | `review-reviewers.yaml` | `47 * * * *` | Hourly analysis of adopter repo sessions | +| `review-reviewers` | `review-reviewers.yaml` | `47 */3 * * *` | Every-3-hours analysis of adopter repo sessions | These use the tend composite action and produce `claude-session-logs*` artifacts, but their names don't match the `tend-*` prefix that scripts filter on by diff --git a/.github/workflows/review-reviewers.yaml b/.github/workflows/review-reviewers.yaml index 7f4e15f5..fc2953dd 100644 --- a/.github/workflows/review-reviewers.yaml +++ b/.github/workflows/review-reviewers.yaml @@ -1,5 +1,13 @@ name: review-reviewers -# Hourly outcome-based analysis of bot behavior across adopter repos. +# Every-3-hours outcome-based analysis of bot behavior across adopter repos. +# +# Cadence is a quota decision, not an analytical one. At hourly × a 5-repo +# matrix this workflow issued 120 Claude sessions/day and took ~70-79% of +# fleet spend, saturating the subscription quota and hard-failing the +# event-driven workflows (review, mention, triage, nightly) that serve users. +# Outcome-acceptance signal — was a bot PR merged, was a comment answered — +# evolves over hours to days, so 3-hourly loses little. `list-recent-runs.sh` +# anchors its window to the cron period, so the tiling holds at either rate. # # Each matrix entry is a repo to analyze. The job checks out the tend repo # (for creating improvement PRs/issues). TEND_BOT_TOKEN is sufficient for reading @@ -9,7 +17,7 @@ name: review-reviewers # TODO: Cross-repo pattern detection — correlate findings across repos on: schedule: - - cron: '47 * * * *' + - cron: '47 */3 * * *' workflow_dispatch: jobs: diff --git a/plugins/tend-ci-runner/scripts/list-recent-runs.sh b/plugins/tend-ci-runner/scripts/list-recent-runs.sh index c16ffa30..6f536181 100755 --- a/plugins/tend-ci-runner/scripts/list-recent-runs.sh +++ b/plugins/tend-ci-runner/scripts/list-recent-runs.sh @@ -1,23 +1,27 @@ #!/usr/bin/env bash # Lists recently completed tend CI runs. # -# Fetches runs started in the past 3 hours, then filters to only those that -# are completed and whose updatedAt falls within a 1-hour completion window. -# This two-step approach is needed because `gh run list --created` filters -# by *start* time, not *end* time — a run started 2h ago may have just -# finished, and a run started 50min ago may still be running. +# Fetches runs started since two hours before the completion window's floor, +# then filters to only those that are completed and whose updatedAt falls +# within that window. This two-step approach is needed because +# `gh run list --created` filters by *start* time, not *end* time — a run +# started 2h ago may have just finished, and a run started 50min ago may +# still be running. # -# Window anchor: when invoked under a scheduled workflow with a simple -# hourly cron (`MM * * * *`), the completion window is anchored to the most -# recent intended cron tick instead of `now`. Consecutive cycles then tile -# exactly: [intended-1h, intended], then [intended, intended+1h]. Without -# this, GHA scheduler delay (20-40 min during peak hours) shifts each -# cycle's window relative to actual start time and drops runs that finished -# in the slack between consecutive actual starts. When GHA *drops* a tick -# entirely (not just delays it), the window's floor is instead pulled back to -# the previous actual run's intended tick so the orphaned hour still gets -# analyzed. For non-schedule events or non-hourly crons, falls back to a -# now-anchored 1h window. +# Window anchor: when invoked under a scheduled workflow with a fixed-period +# cron — hourly (`MM * * * *`) or an every-N-hours step (`MM */N * * *`, N +# dividing 24) — the completion window is anchored to the most recent intended +# cron tick instead of `now`, and is one cron period wide. Consecutive cycles +# then tile exactly: [intended-period, intended], then [intended, +# intended+period]. Without this, GHA scheduler delay (20-40 min during peak +# hours) shifts each cycle's window relative to actual start time and drops +# runs that finished in the slack between consecutive actual starts. When GHA +# *drops* a tick entirely (not just delays it), the window's floor is instead +# pulled back to the previous actual run's intended tick so the orphaned period +# still gets analyzed. For non-schedule events or cron shapes with no constant +# period, falls back to a now-anchored 1h window. Either way the window's floor +# is printed to stderr as `Completion window: >= `, since callers +# filter on it and can't recover it from the run list. # # Environment variables: # TARGET_REPO - Query a different repo (default: current repo) @@ -76,26 +80,44 @@ for prefix in "${PREFIXES[@]}"; do WORKFLOWS+=("${matches[@]}") done -# Detect a simple hourly cron (e.g. "47 * * * *") from the workflow event -# payload so we can anchor the window to the most recent intended tick. +# Detect a fixed-period cron from the workflow event payload so we can anchor +# the window to the most recent intended tick: either hourly ("47 * * * *") or +# an every-N-hours step ("47 */3 * * *"). +# +# The step form is only accepted when N divides 24. Cron's `*/N` restarts the +# count at hour 0 each day, so a step that doesn't divide evenly (e.g. `*/5` +# fires at 0,5,10,15,20 then wraps after 4h) has no constant period, and a +# floor computed from one would silently under-reach across midnight. Those +# fall through to the now-anchored window below, same as any other cron shape. cron_minute="" +cron_hour_step=1 if [ -f "${GITHUB_EVENT_PATH:-}" ]; then schedule=$(jq -r '.schedule // empty' "$GITHUB_EVENT_PATH" 2>/dev/null || true) if [[ "$schedule" =~ ^([0-9]+)\ \*\ \*\ \*\ \*$ ]]; then cron_minute="${BASH_REMATCH[1]}" + elif [[ "$schedule" =~ ^([0-9]+)\ \*/([0-9]+)\ \*\ \*\ \*$ ]] \ + && [ "${BASH_REMATCH[2]}" -gt 0 ] && [ $((24 % BASH_REMATCH[2])) -eq 0 ]; then + cron_minute="${BASH_REMATCH[1]}" + cron_hour_step="${BASH_REMATCH[2]}" fi fi if [ -n "$cron_minute" ]; then - this_hour_tick=$(date -u -d "$(date -u +%Y-%m-%dT%H:00:00) $cron_minute minutes" +%s) + # One cron period in seconds. `cron_hour_step` is 1 for the hourly form, so + # every expression below reduces to the original hourly arithmetic. + period=$((cron_hour_step * 3600)) + # Hour-of-day of the most recent tick: the largest multiple of the step at or + # before the current hour. For the hourly form this is just the current hour. + this_tick_hour=$(( ($(date -u +%-H) / cron_hour_step) * cron_hour_step )) + this_hour_tick=$(date -u -d "$(date -u +%Y-%m-%d)T$(printf '%02d' "$this_tick_hour"):00:00 $cron_minute minutes" +%s) now_ts=$(date -u +%s) if [ "$now_ts" -lt "$this_hour_tick" ]; then - intended=$((this_hour_tick - 3600)) + intended=$((this_hour_tick - period)) else intended=$this_hour_tick fi # Default floor: one cron period back. Consecutive ticks tile exactly. - COMPLETED_AFTER=$((intended - 3600)) + COMPLETED_AFTER=$((intended - period)) # Dropped-tick recovery. GHA doesn't only *delay* scheduled ticks, it also # *drops* them: a tick that fires zero times leaves that hour's completions @@ -104,11 +126,13 @@ if [ -n "$cron_minute" ]; then # fired, resume from where the previous *actual* completed run of this # workflow left off: recover that run's intended tick and floor the window # there. When every tick fires, the previous run's intended tick == the - # default (intended - 3600), so this is a byte-identical no-op — still no + # default (intended - period), so this is a byte-identical no-op — still no # overlap between consecutive cycles. When a tick was dropped, it reaches - # back to cover the orphaned hour. Capped at 6h so a sustained outage can't - # create an unbounded window. The analyzing workflow runs on the current - # repo, so this query omits TARGET_REPO's -R. + # back to cover the orphaned period. Capped at 6h so a sustained outage can't + # create an unbounded window — six full periods at hourly, and still two at + # the 3-hourly step, so a single dropped tick stays recoverable either way. + # The analyzing workflow runs on the current repo, so this query omits + # TARGET_REPO's -R. if [ -n "${GITHUB_WORKFLOW:-}" ]; then prev_start=$(gh run list --workflow "$GITHUB_WORKFLOW" --status completed \ --limit 10 --json databaseId,createdAt \ @@ -117,11 +141,12 @@ if [ -n "$cron_minute" ]; then if [ -n "$prev_start" ]; then prev_ts=$(date -u -d "$prev_start" +%s 2>/dev/null || echo "") if [ -n "$prev_ts" ]; then - prev_hour_tick=$(date -u -d "$(date -u -d "@$prev_ts" +%Y-%m-%dT%H:00:00) $cron_minute minutes" +%s) + prev_tick_hour=$(( ($(date -u -d "@$prev_ts" +%-H) / cron_hour_step) * cron_hour_step )) + prev_hour_tick=$(date -u -d "$(date -u -d "@$prev_ts" +%Y-%m-%d)T$(printf '%02d' "$prev_tick_hour"):00:00 $cron_minute minutes" +%s) if [ "$prev_ts" -ge "$prev_hour_tick" ]; then prev_intended=$prev_hour_tick else - prev_intended=$((prev_hour_tick - 3600)) + prev_intended=$((prev_hour_tick - period)) fi floor_cap=$((intended - 21600)) # never reach back more than 6h [ "$prev_intended" -lt "$floor_cap" ] && prev_intended=$floor_cap @@ -136,6 +161,12 @@ else COMPLETED_AFTER=$(date -d '1 hour ago' +%s) fi +# Publish the window. Callers need the floor for their own time filtering, and +# without it a fallback-branch run (any non-schedule event, e.g. a manual +# workflow_dispatch) looks the same as a scheduled one while covering only the +# last hour of a possibly longer period. stderr keeps stdout the run-list JSON. +echo "Completion window: >= $(date -u -d "@$COMPLETED_AFTER" +%Y-%m-%dT%H:%M:%SZ)" >&2 + all_runs="[]" for wf in "${WORKFLOWS[@]}"; do diff --git a/plugins/tend-ci-runner/skills/review-reviewers/SKILL.md b/plugins/tend-ci-runner/skills/review-reviewers/SKILL.md index c10e54e8..06216c95 100644 --- a/plugins/tend-ci-runner/skills/review-reviewers/SKILL.md +++ b/plugins/tend-ci-runner/skills/review-reviewers/SKILL.md @@ -1,6 +1,6 @@ --- name: review-reviewers -description: Hourly outcome-based analysis of tend's CI behavior — checks whether tend's outputs were accepted or rejected, escalating to session logs only when outcomes look wrong. +description: Scheduled outcome-based analysis of tend's CI behavior — checks whether tend's outputs were accepted or rejected, escalating to session logs only when outcomes look wrong. argument-hint: "" metadata: internal: true @@ -8,7 +8,7 @@ metadata: # Review Reviewers -Analyze tend's CI behavior on the target repo over the past hour. Focus on **outcomes** — what the bot produced publicly and whether it was accepted — rather than internal session mechanics. Create PRs or issues on tend when outcomes reveal behavioral problems. +Analyze tend's CI behavior on the target repo over the analysis window — the run list in Step 1 is anchored to the workflow's cron period, so take its span as the window rather than assuming any fixed number of hours. Focus on **outcomes** — what the bot produced publicly and whether it was accepted — rather than internal session mechanics. Create PRs or issues on tend when outcomes reveal behavioral problems. ## First steps @@ -213,6 +213,8 @@ TARGET_REPO=$ARGUMENTS ${CLAUDE_PLUGIN_ROOT}/scripts/list-recent-runs.sh The script discovers `tend-*` workflows by default. Pass additional prefixes as arguments to include other workflows (e.g., `review-reviewers` when analyzing tend itself). +It prints `Completion window: >= ` on stderr. **Note that timestamp — it is the analysis window's floor**, and Step 2 needs it both for the survey prompt and for the corruption scan's `--jq` filters; nothing downstream can reconstruct it from the run list. The window is normally one cron period wide on a scheduled run — wider when the script recovers a dropped tick — but only one hour on any other event (a manual `workflow_dispatch` takes that path), so on a dispatched run expect it to cover less than a full period. + If empty, record the run as all-clear per "Recording below-threshold findings" above, then skip to Step 6. ## Step 2: Survey outcomes via cheap subagent @@ -222,7 +224,9 @@ Spawn a cheap subagent to check outcomes across all runs from Step 1. The subage Use a cheap subagent (e.g. Haiku / gpt-mini) and a prompt like: > Survey bot outcomes on `$ARGUMENTS` for the following runs: [run IDs from Step 1]. -> The bot's login is `$BOT_LOGIN`. +> The bot's login is `$BOT_LOGIN`. The analysis window starts at [completion-window +> floor from Step 1] and runs to now — substitute that timestamp everywhere +> `` appears below. > > For each run, determine: > 1. Did the bot produce visible output (review, comment, issue action, commit)? @@ -256,7 +260,7 @@ Use a cheap subagent (e.g. Haiku / gpt-mini) and a prompt like: > - `tend-review`: `gh -R $ARGUMENTS run view --json headBranch` → find PR via > `gh -R $ARGUMENTS pr list --head --state all` → check bot reviews via > `gh api repos/$ARGUMENTS/pulls//reviews` -> - `tend-notifications`: check for recent bot comments/issue-close events in the past hour +> - `tend-notifications`: check for bot comments/issue-close events since the window start above (this mapping has no run-scoped anchor, so it reconstructs outcomes from the window alone — don't substitute a fixed hour for it) > - `tend-mention`: map run to issue/PR from triggering comment, check for bot replies > - `tend-ci-fix`: map run → PR via `headBranch`, check for bot commits > @@ -382,7 +386,7 @@ Search titles AND bodies for related keywords. Only comment on existing issues i **Prefer PRs over issues.** A PR with a clear description is immediately actionable. -- **PR** (default): Branch `hourly/review-$GITHUB_RUN_ID--`, fix, commit, push, create with label `claude-behavior`. `$GITHUB_RUN_ID` alone is not a unique branch name: every matrix leg of a tick carries the same one, and a single leg may open two PRs (see the 2-PR limit below). The target's repo name (the part after the `/`) keeps two legs from racing the same ref; the topic slug keeps one leg's two PRs from doing the same. Put full analysis in PR description (run ID, outcome evidence, root cause, **gate assessment** including historical evidence count). Don't also create a separate issue. +- **PR** (default): Branch `review-reviewers/review-$GITHUB_RUN_ID--`, fix, commit, push, create with label `claude-behavior`. `$GITHUB_RUN_ID` alone is not a unique branch name: every matrix leg of a tick carries the same one, and a single leg may open two PRs (see the 2-PR limit below). The target's repo name (the part after the `/`) keeps two legs from racing the same ref; the topic slug keeps one leg's two PRs from doing the same. Put full analysis in PR description (run ID, outcome evidence, root cause, **gate assessment** including historical evidence count). Don't also create a separate issue. - **Issue** (fallback): Only for problems too large or ambiguous to fix directly. Include run ID, outcome evidence, root cause analysis. Group multiple findings by broad theme. **Limit to at most 2 PRs per run** — if you have more findings, pick the highest-confidence ones and record the rest in the evidence gist.