Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude/skills/running-tend/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Tend has Claude-powered workflows beyond the generated `tend-*` set:

| Workflow | File | Schedule | Purpose |
|----------|------|----------|---------|
| `review-reviewers` | `review-reviewers.yaml` | `47 * * * *` | Hourly analysis of adopter repo sessions |
| `review-reviewers` | `review-reviewers.yaml` | `47 */3 * * *` | Every-3-hours analysis of adopter repo sessions |

These use the tend composite action and produce `claude-session-logs*` artifacts,
but their names don't match the `tend-*` prefix that scripts filter on by
Expand Down
12 changes: 10 additions & 2 deletions .github/workflows/review-reviewers.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@
name: review-reviewers
# Hourly outcome-based analysis of bot behavior across adopter repos.
# Every-3-hours outcome-based analysis of bot behavior across adopter repos.
#
# Cadence is a quota decision, not an analytical one. At hourly × a 5-repo
# matrix this workflow issued 120 Claude sessions/day and took ~70-79% of
# fleet spend, saturating the subscription quota and hard-failing the
# event-driven workflows (review, mention, triage, nightly) that serve users.
# Outcome-acceptance signal — was a bot PR merged, was a comment answered —
# evolves over hours to days, so 3-hourly loses little. `list-recent-runs.sh`
# anchors its window to the cron period, so the tiling holds at either rate.
#
# Each matrix entry is a repo to analyze. The job checks out the tend repo
# (for creating improvement PRs/issues). TEND_BOT_TOKEN is sufficient for reading
Expand All @@ -9,7 +17,7 @@ name: review-reviewers
# TODO: Cross-repo pattern detection — correlate findings across repos
on:
schedule:
- cron: '47 * * * *'
- cron: '47 */3 * * *'
workflow_dispatch:

jobs:
Expand Down
85 changes: 58 additions & 27 deletions plugins/tend-ci-runner/scripts/list-recent-runs.sh
Original file line number Diff line number Diff line change
@@ -1,23 +1,27 @@
#!/usr/bin/env bash
# Lists recently completed tend CI runs.
#
# Fetches runs started in the past 3 hours, then filters to only those that
# are completed and whose updatedAt falls within a 1-hour completion window.
# This two-step approach is needed because `gh run list --created` filters
# by *start* time, not *end* time — a run started 2h ago may have just
# finished, and a run started 50min ago may still be running.
# Fetches runs started since two hours before the completion window's floor,
# then filters to only those that are completed and whose updatedAt falls
# within that window. This two-step approach is needed because
# `gh run list --created` filters by *start* time, not *end* time — a run
# started 2h ago may have just finished, and a run started 50min ago may
# still be running.
#
# Window anchor: when invoked under a scheduled workflow with a simple
# hourly cron (`MM * * * *`), the completion window is anchored to the most
# recent intended cron tick instead of `now`. Consecutive cycles then tile
# exactly: [intended-1h, intended], then [intended, intended+1h]. Without
# this, GHA scheduler delay (20-40 min during peak hours) shifts each
# cycle's window relative to actual start time and drops runs that finished
# in the slack between consecutive actual starts. When GHA *drops* a tick
# entirely (not just delays it), the window's floor is instead pulled back to
# the previous actual run's intended tick so the orphaned hour still gets
# analyzed. For non-schedule events or non-hourly crons, falls back to a
# now-anchored 1h window.
# Window anchor: when invoked under a scheduled workflow with a fixed-period
# cron — hourly (`MM * * * *`) or an every-N-hours step (`MM */N * * *`, N
# dividing 24) — the completion window is anchored to the most recent intended
# cron tick instead of `now`, and is one cron period wide. Consecutive cycles
# then tile exactly: [intended-period, intended], then [intended,
# intended+period]. Without this, GHA scheduler delay (20-40 min during peak
# hours) shifts each cycle's window relative to actual start time and drops
# runs that finished in the slack between consecutive actual starts. When GHA
# *drops* a tick entirely (not just delays it), the window's floor is instead
# pulled back to the previous actual run's intended tick so the orphaned period
# still gets analyzed. For non-schedule events or cron shapes with no constant
# period, falls back to a now-anchored 1h window. Either way the window's floor
# is printed to stderr as `Completion window: >= <timestamp>`, since callers
# filter on it and can't recover it from the run list.
#
# Environment variables:
# TARGET_REPO - Query a different repo (default: current repo)
Expand Down Expand Up @@ -76,26 +80,44 @@ for prefix in "${PREFIXES[@]}"; do
WORKFLOWS+=("${matches[@]}")
done

# Detect a simple hourly cron (e.g. "47 * * * *") from the workflow event
# payload so we can anchor the window to the most recent intended tick.
# Detect a fixed-period cron from the workflow event payload so we can anchor
# the window to the most recent intended tick: either hourly ("47 * * * *") or
# an every-N-hours step ("47 */3 * * *").
#
# The step form is only accepted when N divides 24. Cron's `*/N` restarts the
# count at hour 0 each day, so a step that doesn't divide evenly (e.g. `*/5`
# fires at 0,5,10,15,20 then wraps after 4h) has no constant period, and a
# floor computed from one would silently under-reach across midnight. Those
# fall through to the now-anchored window below, same as any other cron shape.
cron_minute=""
cron_hour_step=1
if [ -f "${GITHUB_EVENT_PATH:-}" ]; then
schedule=$(jq -r '.schedule // empty' "$GITHUB_EVENT_PATH" 2>/dev/null || true)
if [[ "$schedule" =~ ^([0-9]+)\ \*\ \*\ \*\ \*$ ]]; then
cron_minute="${BASH_REMATCH[1]}"
elif [[ "$schedule" =~ ^([0-9]+)\ \*/([0-9]+)\ \*\ \*\ \*$ ]] \
&& [ "${BASH_REMATCH[2]}" -gt 0 ] && [ $((24 % BASH_REMATCH[2])) -eq 0 ]; then
cron_minute="${BASH_REMATCH[1]}"
cron_hour_step="${BASH_REMATCH[2]}"
fi
fi

if [ -n "$cron_minute" ]; then
this_hour_tick=$(date -u -d "$(date -u +%Y-%m-%dT%H:00:00) $cron_minute minutes" +%s)
# One cron period in seconds. `cron_hour_step` is 1 for the hourly form, so
# every expression below reduces to the original hourly arithmetic.
period=$((cron_hour_step * 3600))
# Hour-of-day of the most recent tick: the largest multiple of the step at or
# before the current hour. For the hourly form this is just the current hour.
this_tick_hour=$(( ($(date -u +%-H) / cron_hour_step) * cron_hour_step ))
this_hour_tick=$(date -u -d "$(date -u +%Y-%m-%d)T$(printf '%02d' "$this_tick_hour"):00:00 $cron_minute minutes" +%s)
now_ts=$(date -u +%s)
if [ "$now_ts" -lt "$this_hour_tick" ]; then
intended=$((this_hour_tick - 3600))
intended=$((this_hour_tick - period))
else
intended=$this_hour_tick
fi
# Default floor: one cron period back. Consecutive ticks tile exactly.
COMPLETED_AFTER=$((intended - 3600))
COMPLETED_AFTER=$((intended - period))

# Dropped-tick recovery. GHA doesn't only *delay* scheduled ticks, it also
# *drops* them: a tick that fires zero times leaves that hour's completions
Expand All @@ -104,11 +126,13 @@ if [ -n "$cron_minute" ]; then
# fired, resume from where the previous *actual* completed run of this
# workflow left off: recover that run's intended tick and floor the window
# there. When every tick fires, the previous run's intended tick == the
# default (intended - 3600), so this is a byte-identical no-op — still no
# default (intended - period), so this is a byte-identical no-op — still no
# overlap between consecutive cycles. When a tick was dropped, it reaches
# back to cover the orphaned hour. Capped at 6h so a sustained outage can't
# create an unbounded window. The analyzing workflow runs on the current
# repo, so this query omits TARGET_REPO's -R.
# back to cover the orphaned period. Capped at 6h so a sustained outage can't
# create an unbounded window — six full periods at hourly, and still two at
# the 3-hourly step, so a single dropped tick stays recoverable either way.
# The analyzing workflow runs on the current repo, so this query omits
# TARGET_REPO's -R.
if [ -n "${GITHUB_WORKFLOW:-}" ]; then
prev_start=$(gh run list --workflow "$GITHUB_WORKFLOW" --status completed \
--limit 10 --json databaseId,createdAt \
Expand All @@ -117,11 +141,12 @@ if [ -n "$cron_minute" ]; then
if [ -n "$prev_start" ]; then
prev_ts=$(date -u -d "$prev_start" +%s 2>/dev/null || echo "")
if [ -n "$prev_ts" ]; then
prev_hour_tick=$(date -u -d "$(date -u -d "@$prev_ts" +%Y-%m-%dT%H:00:00) $cron_minute minutes" +%s)
prev_tick_hour=$(( ($(date -u -d "@$prev_ts" +%-H) / cron_hour_step) * cron_hour_step ))
prev_hour_tick=$(date -u -d "$(date -u -d "@$prev_ts" +%Y-%m-%d)T$(printf '%02d' "$prev_tick_hour"):00:00 $cron_minute minutes" +%s)
if [ "$prev_ts" -ge "$prev_hour_tick" ]; then
prev_intended=$prev_hour_tick
else
prev_intended=$((prev_hour_tick - 3600))
prev_intended=$((prev_hour_tick - period))
fi
floor_cap=$((intended - 21600)) # never reach back more than 6h
[ "$prev_intended" -lt "$floor_cap" ] && prev_intended=$floor_cap
Expand All @@ -136,6 +161,12 @@ else
COMPLETED_AFTER=$(date -d '1 hour ago' +%s)
fi

# Publish the window. Callers need the floor for their own time filtering, and
# without it a fallback-branch run (any non-schedule event, e.g. a manual
# workflow_dispatch) looks the same as a scheduled one while covering only the
# last hour of a possibly longer period. stderr keeps stdout the run-list JSON.
echo "Completion window: >= $(date -u -d "@$COMPLETED_AFTER" +%Y-%m-%dT%H:%M:%SZ)" >&2

all_runs="[]"

for wf in "${WORKFLOWS[@]}"; do
Expand Down
14 changes: 9 additions & 5 deletions plugins/tend-ci-runner/skills/review-reviewers/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
---
name: review-reviewers
description: Hourly outcome-based analysis of tend's CI behavior — checks whether tend's outputs were accepted or rejected, escalating to session logs only when outcomes look wrong.
description: Scheduled outcome-based analysis of tend's CI behavior — checks whether tend's outputs were accepted or rejected, escalating to session logs only when outcomes look wrong.
argument-hint: "<owner/repo>"
metadata:
internal: true
---

# Review Reviewers

Analyze tend's CI behavior on the target repo over the past hour. Focus on **outcomes** — what the bot produced publicly and whether it was accepted — rather than internal session mechanics. Create PRs or issues on tend when outcomes reveal behavioral problems.
Analyze tend's CI behavior on the target repo over the analysis window — the run list in Step 1 is anchored to the workflow's cron period, so take its span as the window rather than assuming any fixed number of hours. Focus on **outcomes** — what the bot produced publicly and whether it was accepted — rather than internal session mechanics. Create PRs or issues on tend when outcomes reveal behavioral problems.

## First steps

Expand Down Expand Up @@ -213,6 +213,8 @@ TARGET_REPO=$ARGUMENTS ${CLAUDE_PLUGIN_ROOT}/scripts/list-recent-runs.sh

The script discovers `tend-*` workflows by default. Pass additional prefixes as arguments to include other workflows (e.g., `review-reviewers` when analyzing tend itself).

It prints `Completion window: >= <timestamp>` on stderr. **Note that timestamp — it is the analysis window's floor**, and Step 2 needs it both for the survey prompt and for the corruption scan's `--jq` filters; nothing downstream can reconstruct it from the run list. The window is normally one cron period wide on a scheduled run — wider when the script recovers a dropped tick — but only one hour on any other event (a manual `workflow_dispatch` takes that path), so on a dispatched run expect it to cover less than a full period.

If empty, record the run as all-clear per "Recording below-threshold findings" above, then skip to Step 6.

## Step 2: Survey outcomes via cheap subagent
Expand All @@ -222,7 +224,9 @@ Spawn a cheap subagent to check outcomes across all runs from Step 1. The subage
Use a cheap subagent (e.g. Haiku / gpt-mini) and a prompt like:

> Survey bot outcomes on `$ARGUMENTS` for the following runs: [run IDs from Step 1].
> The bot's login is `$BOT_LOGIN`.
> The bot's login is `$BOT_LOGIN`. The analysis window starts at [completion-window
> floor from Step 1] and runs to now — substitute that timestamp everywhere
> `<window-start>` appears below.
>
> For each run, determine:
> 1. Did the bot produce visible output (review, comment, issue action, commit)?
Expand Down Expand Up @@ -256,7 +260,7 @@ Use a cheap subagent (e.g. Haiku / gpt-mini) and a prompt like:
> - `tend-review`: `gh -R $ARGUMENTS run view <run-id> --json headBranch` → find PR via
> `gh -R $ARGUMENTS pr list --head <branch> --state all` → check bot reviews via
> `gh api repos/$ARGUMENTS/pulls/<pr>/reviews`
> - `tend-notifications`: check for recent bot comments/issue-close events in the past hour
> - `tend-notifications`: check for bot comments/issue-close events since the window start above (this mapping has no run-scoped anchor, so it reconstructs outcomes from the window alone — don't substitute a fixed hour for it)
> - `tend-mention`: map run to issue/PR from triggering comment, check for bot replies
> - `tend-ci-fix`: map run → PR via `headBranch`, check for bot commits
>
Expand Down Expand Up @@ -382,7 +386,7 @@ Search titles AND bodies for related keywords. Only comment on existing issues i

**Prefer PRs over issues.** A PR with a clear description is immediately actionable.

- **PR** (default): Branch `hourly/review-$GITHUB_RUN_ID-<target-repo-name>-<topic-slug>`, fix, commit, push, create with label `claude-behavior`. `$GITHUB_RUN_ID` alone is not a unique branch name: every matrix leg of a tick carries the same one, and a single leg may open two PRs (see the 2-PR limit below). The target's repo name (the part after the `/`) keeps two legs from racing the same ref; the topic slug keeps one leg's two PRs from doing the same. Put full analysis in PR description (run ID, outcome evidence, root cause, **gate assessment** including historical evidence count). Don't also create a separate issue.
- **PR** (default): Branch `review-reviewers/review-$GITHUB_RUN_ID-<target-repo-name>-<topic-slug>`, fix, commit, push, create with label `claude-behavior`. `$GITHUB_RUN_ID` alone is not a unique branch name: every matrix leg of a tick carries the same one, and a single leg may open two PRs (see the 2-PR limit below). The target's repo name (the part after the `/`) keeps two legs from racing the same ref; the topic slug keeps one leg's two PRs from doing the same. Put full analysis in PR description (run ID, outcome evidence, root cause, **gate assessment** including historical evidence count). Don't also create a separate issue.
- **Issue** (fallback): Only for problems too large or ambiguous to fix directly. Include run ID, outcome evidence, root cause analysis.

Group multiple findings by broad theme. **Limit to at most 2 PRs per run** — if you have more findings, pick the highest-confidence ones and record the rest in the evidence gist.
Expand Down
Loading