Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 15 additions & 19 deletions .github/workflows/wporg-investigate-job.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ on:
workflow_call:
inputs:
matrix:
description: "JSON array of {topic_url, slug, repo, kind, fallback_repo} entries"
description: "JSON array of {topic_url, slug, repo (free, code is read), kind, issue_repo (pro if the product has one, else free)} entries"
required: true
type: string
model:
Expand Down Expand Up @@ -40,31 +40,27 @@ jobs:
token: ${{ startsWith(matrix.repo, 'themegrill/') && secrets.BOT_TOKEN_THEMEGRILL || secrets.BOT_TOKEN }}
path: workspace

# Where issues get filed. Normally matrix.repo; if the bot token cannot
# write issues there (a fine-grained PAT only reaches the repos it was
# granted), use matrix.fallback_repo instead. Probe = create the QA
# labels the agent needs anyway: 201 (created) or 422 (exists) means
# write access; anything else means no.
- name: Choose the issue repo
# Preflight on matrix.issue_repo (the -pro repo when the product has one,
# otherwise the free repo): fail loudly BEFORE spending an agent run if the
# token cannot write issues there. A fine-grained PAT only reaches the
# repos it was granted. Probe = create the QA label the agent needs
# anyway: 201 (created) or 422 (exists) means write access. There is
# deliberately no silent fallback to another repo.
- name: Check write access to the issue repo
id: target
env:
PRIMARY: ${{ matrix.repo }}
FALLBACK: ${{ matrix.fallback_repo }}
GH_TOKEN: ${{ startsWith(matrix.repo, 'themegrill/') && secrets.BOT_TOKEN_THEMEGRILL || secrets.BOT_TOKEN }}
ISSUE_REPO: ${{ matrix.issue_repo }}
GH_TOKEN: ${{ startsWith(matrix.issue_repo, 'themegrill/') && secrets.BOT_TOKEN_THEMEGRILL || secrets.BOT_TOKEN }}
run: |
code=$(gh api -X POST "repos/$PRIMARY/labels" -f name=qa-verified -f color=0e8a16 \
code=$(gh api -X POST "repos/$ISSUE_REPO/labels" -f name=qa-verified -f color=0e8a16 \
-f description="AI-filed issue reproduced directly with high confidence" \
--include 2>/dev/null | head -1 | awk '{print $2}')
echo "primary label probe on $PRIMARY: HTTP ${code:-none}"
if [ "$code" = "201" ] || [ "$code" = "422" ]; then
echo "repo=$PRIMARY" >> "$GITHUB_OUTPUT"
elif [ -n "$FALLBACK" ]; then
echo "::warning::Token cannot write issues to $PRIMARY (HTTP ${code:-none}); filing in $FALLBACK instead."
echo "repo=$FALLBACK" >> "$GITHUB_OUTPUT"
else
echo "::error::Token cannot write issues to $PRIMARY (HTTP ${code:-none}) and no fallback_repo is configured."
echo "label probe on $ISSUE_REPO: HTTP ${code:-none}"
if [ "$code" != "201" ] && [ "$code" != "422" ]; then
echo "::error::Token cannot write issues to $ISSUE_REPO (HTTP ${code:-none}). Check the PAT's repository access; re-run via manual dispatch afterwards."
exit 1
fi
echo "repo=$ISSUE_REPO" >> "$GITHUB_OUTPUT"

- uses: actions/setup-node@v4
with:
Expand Down
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

Short, dated summary of notable fixes and changes. For the full "why," see `PHASE2-SETUP.md` (Crisp triage design) or the linked PRs.

## 2026-10-05 — wp.org triage files issues in the pro repo first

Issues now go to the product's `-pro` repo when it has one, otherwise to the free repo (`issue_repos` in `wporg/config/products.json`, replacing `fallback_repos`). The agent still reads the free repo's code and searches both repos for existing issues (Crisp files free-edition issues in the free repos, so a match can live there). The old "probe the free repo, fall back to pro" behavior is gone: the job now only checks that the token can write to the chosen issue repo and fails loudly if not (the topic is already marked seen, so re-run it by hand).

## 2026-10-05 — wp.org triage: filing threshold, closed-issue search, `wporg-forum` label

Interim policy after two false-positive issues on the same topic (#1243, #1244, both closed): the agent files only when its own confidence is above 50; below 70 or unreproduced it gets `manual-qa-required` (unchanged). Duplicate search now uses `--state all` so a match that a human already closed is not re-filed or commented on. Every filed issue also gets a `wporg-forum` label for bulk review, and the issue body gains a "Possible fix" section (suggestion only; "No fix proposed" when the diagnosis does not support one). Issues with confidence below 70 begin with a fixed "Low-confidence AI diagnosis -- verify manually" warning. Caveat: self-reported confidence is not calibrated (94 then 60 for the same wrong claim), so this does not stop a confidently wrong diagnosis; review the first issues by hand.
Expand Down
2 changes: 1 addition & 1 deletion wporg/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ flowchart LR
- Only **new** topics are looked at. The first run for a product only records what already exists (`seeded`), so onboarding never floods Stage 2 with a backlog. **Resolved topics are skipped** (read from the forum list page's resolved marker; if that page can't be fetched, nothing is treated as resolved). Topics created more than 14 days ago (`MAX_AGE_DAYS`, by creation date, not last reply) that were never seen are marked seen and skipped. At most 10 escalations per run; the rest are marked seen and can be run manually.
- Same rules as Crisp Stage 2: bug and feature judged separately, existing issue -> comment (a feature needs an exact-capability match), otherwise new issue with `manual-qa-required` / `qa-verified`. Only a `product_bug` root cause may be filed; `conflict` / `host` / `user_error` / `undetermined` file nothing.
- **Nothing is ever written to wordpress.org** (no official API, and bot replies there risk trouble with the plugin team). The agent's report goes to the job log / step summary only. There is no draft reply.
- **Issue repo fallback**: the bot token (a fine-grained PAT) only reaches the repos it was granted. Before the agent runs, the job probes write access to the free repo (by creating the QA labels). If that fails, issues are filed in the product's `-pro` repo instead (`fallback_repos` in `config/products.json`, derived from `config/inbox-to-repo.json`), with that org's token, and the run logs a warning. Products with no pro repo fail loudly. Pro repos are private, so a fallback issue is only visible to the team. Fixing the PAT's repository list makes the fallback unnecessary.
- **Where issues are filed**: in the product's `-pro` repo when it has one, otherwise in the free repo (`issue_repos` in `config/products.json`, derived from `config/inbox-to-repo.json`). The agent still reads the **free** repo's code, since the forum is about the free edition, and searches both repos for existing issues. Pro repos are private, so only the team sees these issues. Before the agent runs, the job checks that the bot token can write issues to that repo (by creating the QA label); if not it fails loudly, with no silent fallback to another repo. Re-run the topic with the manual dispatch after fixing the PAT's repository access.
- Issues link back with `Source: [WordPress.org forum topic](url)` and must not copy usernames, emails, site URLs or license keys.

## Trust
Expand Down
4 changes: 2 additions & 2 deletions wporg/config/products.json
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@
"suffice": "themegrill/suffice",
"zakra": "themegrill/zakra"
},
"_fallback_repos": "Used ONLY when the bot token cannot write issues to the free repo (probed at run time by wporg-triage). Free repo -> its -pro repo, derived from config/inbox-to-repo.json. Pro repos are private; the issue only describes public forum content and free-edition code.",
"fallback_repos": {
"_issue_repos": "Where GitHub issues are FILED for a product's forum topics. wp.org only hosts the free edition, so the agent reads the free repo's code, but issues go to the -pro repo when the product has one (derived from config/inbox-to-repo.json); products with no pro repo file in the free repo. Pro repos are private, so only the team sees these issues.",
"issue_repos": {
"wpeverest/user-registration": "themegrill/user-registration-pro",
"wpeverest/everest-forms": "themegrill/everest-forms-pro",
"themegrill/accelerate": "themegrill/accelerate-pro",
Expand Down
1 change: 1 addition & 0 deletions wporg/prompt.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ Replies marked "Plugin/Theme Support (our team)" are from our own support staff
```
gh issue list --repo {{ISSUE_REPO}} --state all --search "<relevant keywords>"
```
If `{{REPO}}` and `{{ISSUE_REPO}}` are different repos, ALSO run the same search with `--repo {{REPO}}`: issues for the free edition may already be tracked there. A match in either repo counts as a match. Comment on the match in whichever repo holds it; if that comment is refused for lack of permission, do not retry elsewhere and do not file a duplicate: record "Already tracked: <url> (could not comment)" in step 5. New issues are always filed in `{{ISSUE_REPO}}`.
If you find a genuine match, **first check whether that issue's own `Source:` line already references this topic** (`{{CONVERSATION_URL}}`) -- if so, it is not a recurrence; skip commenting and record it as "already tracked". Otherwise comment on the match (`gh issue comment <number> --repo {{ISSUE_REPO}} --body-file <path>`): for a bug, say another user reports the same problem; for a feature request, say another user asked for the same thing. Briefly state what this topic adds and link `[WordPress.org forum topic]({{CONVERSATION_URL}})`. Do not repeat the full diagnosis. Do not file a new issue for something already tracked.

**A closed match counts too.** Searching `--state all` also shows closed issues. If the genuine match is closed, a human already fixed or rejected it (several AI-filed issues were closed as wrong diagnoses): do not comment on it and do not file a new one; record it as "Previously closed: <url>" in step 5.
Expand Down
6 changes: 3 additions & 3 deletions wporg/scripts/wporg-classify.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,14 @@ const state = JSON.parse(await readFile(STATE_PATH, "utf8"));
const now = Date.now();
const matrix = [];

// fallback_repo is "" when none; the workflow probes write access and only
// uses it if the token cannot write issues to `repo`.
// `repo` = the free repo (its code is read); `issue_repo` = where issues are
// filed: the -pro repo when the product has one, otherwise the free repo.
const entry = (topic_url, repo, kind) => ({
topic_url,
slug: topicSlug(topic_url),
repo,
kind,
fallback_repo: config.fallback_repos?.[repo] ?? "",
issue_repo: config.issue_repos?.[repo] ?? repo,
});

async function main() {
Expand Down
Loading