From b2e17548ce59634d86210d255bf130eea32554d3 Mon Sep 17 00:00:00 2001 From: Will Pike <6687499+pike00@users.noreply.github.com> Date: Thu, 7 May 2026 14:28:25 -0500 Subject: [PATCH 1/6] Add sequential numbering and suggested fixes to again skill output Issues now get a single global ID across must_fix/should_fix/suggestion tables so users can reference fixes by number (e.g. 'apply fix 1, 3, 5'). Each table gains a # column and a Suggested Fix column, and aggregate.md mirrors the same columns with full untruncated fix text. --- src/skills/again/SKILL.md | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/src/skills/again/SKILL.md b/src/skills/again/SKILL.md index 83a816e..24090c8 100644 --- a/src/skills/again/SKILL.md +++ b/src/skills/again/SKILL.md @@ -88,6 +88,7 @@ After completing the configured number of passes, if `must_fix` issues remain an 1. **Deduplicate** on (file, title). Same issue across passes = higher confidence. 2. **Score**: Confidence = (passes finding issue) / (total passes) x 100%. 3. **Group** by severity, sort by confidence within groups. +4. **Number**: Assign a single global sequential ID (`1..N`) across all issues. Order: every `must_fix` first (in confidence-sorted order), then every `should_fix`, then every `suggestion`. The same ID identifies the issue in the conversation tables, in `aggregate.md`, and in `aggregate.json` (store as the `id` field on each issue). The user can reference fixes by these numbers (e.g. "apply fix 1, 3, 5"). ## Phase 3: Save and Report @@ -95,6 +96,8 @@ Save to `.lookagain//`: - `pass-N.json` after each pass - `aggregate.json` and `aggregate.md` after aggregation +`aggregate.md` uses the same tables shown below (same columns, same global IDs). Keep `suggested_fix` text intact in the markdown file - do not truncate. + Present the final summary to the user in this format: ``` @@ -103,27 +106,31 @@ Present the final summary to the user in this format: **Passes completed**: N **Unique issues found**: X +Reference fixes by number (e.g. "apply fix 1, 3, 5"). + ### Must Fix (N issues) -| Issue | File | Confidence | Fixed | -| ----- | ---- | ---------- | ----- | -| ... | ... | ...% | Yes/No | +| # | Issue | File | Confidence | Suggested Fix | Fixed | +| - | ----- | ---- | ---------- | ------------- | ----- | +| 1 | ... | ... | ...% | ... | Yes/No | ### Should Fix (N issues) -| Issue | File | Confidence | -| ----- | ---- | ---------- | -| ... | ... | ...% | +| # | Issue | File | Confidence | Suggested Fix | +| - | ----- | ---- | ---------- | ------------- | +| 4 | ... | ... | ...% | ... | ### Suggestions (N issues) -| Issue | File | Confidence | -| ----- | ---- | ---------- | -| ... | ... | ...% | +| # | Issue | File | Confidence | Suggested Fix | +| - | ----- | ---- | ---------- | ------------- | +| 7 | ... | ... | ...% | ... | Full report saved to `.lookagain//aggregate.md` ``` +The `#` column carries the global ID assigned in Phase 2 - numbering does not restart between tables. In conversation output, you may collapse very long `Suggested Fix` text to a single line; the full text always lives in `aggregate.md` and `aggregate.json`. + Include the count of previous runs (glob `.lookagain/????-??-??T??-??-??/`, subtract 1). Mention `/look:tidy` if previous runs exist. ## Rules From e204f7285a797a382d25122c89c75740ad786701 Mon Sep 17 00:00:00 2001 From: Will Pike <6687499+pike00@users.noreply.github.com> Date: Thu, 7 May 2026 15:15:18 -0500 Subject: [PATCH 2/6] Address review: escape pipes/newlines in fix cells, widen separator dashes --- src/skills/again/SKILL.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/src/skills/again/SKILL.md b/src/skills/again/SKILL.md index 24090c8..3b7e8cb 100644 --- a/src/skills/again/SKILL.md +++ b/src/skills/again/SKILL.md @@ -96,7 +96,7 @@ Save to `.lookagain//`: - `pass-N.json` after each pass - `aggregate.json` and `aggregate.md` after aggregation -`aggregate.md` uses the same tables shown below (same columns, same global IDs). Keep `suggested_fix` text intact in the markdown file - do not truncate. +`aggregate.md` uses the same tables shown below (same columns, same global IDs). Keep `suggested_fix` text intact in the markdown file - do not truncate. To preserve the table format, escape pipe characters as `\|` and replace embedded newlines with `
` (or single spaces) inside the `Suggested Fix` cell. Present the final summary to the user in this format: @@ -110,21 +110,21 @@ Reference fixes by number (e.g. "apply fix 1, 3, 5"). ### Must Fix (N issues) -| # | Issue | File | Confidence | Suggested Fix | Fixed | -| - | ----- | ---- | ---------- | ------------- | ----- | -| 1 | ... | ... | ...% | ... | Yes/No | +| # | Issue | File | Confidence | Suggested Fix | Fixed | +| --- | ----- | ---- | ---------- | ------------- | ----- | +| 1 | ... | ... | ...% | ... | Yes/No | ### Should Fix (N issues) -| # | Issue | File | Confidence | Suggested Fix | -| - | ----- | ---- | ---------- | ------------- | -| 4 | ... | ... | ...% | ... | +| # | Issue | File | Confidence | Suggested Fix | +| --- | ----- | ---- | ---------- | ------------- | +| 4 | ... | ... | ...% | ... | ### Suggestions (N issues) -| # | Issue | File | Confidence | Suggested Fix | -| - | ----- | ---- | ---------- | ------------- | -| 7 | ... | ... | ...% | ... | +| # | Issue | File | Confidence | Suggested Fix | +| --- | ----- | ---- | ---------- | ------------- | +| 7 | ... | ... | ...% | ... | Full report saved to `.lookagain//aggregate.md` ``` From d78fcaafd274f8b174bbe3f0fb30d486525a5d78 Mon Sep 17 00:00:00 2001 From: Will Pike <6687499+pike00@users.noreply.github.com> Date: Fri, 8 May 2026 12:26:48 -0500 Subject: [PATCH 3/6] Add global sequential numbering and Suggested Fix column to summary tables --- evals/promptfooconfig.yaml | 44 +++++++++++++++++++++++++++++++++++++- 1 file changed, 43 insertions(+), 1 deletion(-) diff --git a/evals/promptfooconfig.yaml b/evals/promptfooconfig.yaml index af195f7..1c5af1b 100644 --- a/evals/promptfooconfig.yaml +++ b/evals/promptfooconfig.yaml @@ -4,7 +4,7 @@ prompts: - file://prompt-loader.js providers: - - id: anthropic:messages:claude-sonnet-4-20250514 + - id: anthropic:messages:claude-sonnet-4-6 config: max_tokens: 2048 @@ -145,6 +145,48 @@ tests: thorough model, max-passes 7). It must NOT apply any fixes since auto-fix is explicitly false. + # ================================================================== + # again — global sequential numbering + Suggested Fix column + # ================================================================== + - description: "summary tables include Suggested Fix column and global ID" + vars: + prompt_file: src/skills/again/SKILL.md + assert: + - type: icontains + value: "Suggested Fix" + - type: llm-rubric + value: > + The response must describe a final summary that includes three tables + (Must Fix, Should Fix, Suggestions), each containing a "Suggested Fix" + column and a global sequential ID column (often shown as "#"). It must + state that IDs are assigned once across all issues and DO NOT restart + between the three tables (e.g., must_fix uses 1..k, should_fix + continues at k+1, etc.). It must also state that the user can refer + to fixes by these IDs (e.g., "apply fix 1, 3, 5"). + + - description: "aggregate.md preserves full Suggested Fix text; conversation may truncate" + vars: + prompt_file: src/skills/again/SKILL.md + assert: + - type: llm-rubric + value: > + The response must indicate that the full untruncated Suggested Fix + text is preserved in aggregate.md (and aggregate.json), while the + conversation-summary table is allowed to collapse very long Suggested + Fix text to a single line. It must NOT say the full text is dropped + or only kept in conversation. + + - description: "must_fix table includes a Fixed column; should_fix and suggestions do not" + vars: + prompt_file: src/skills/again/SKILL.md + assert: + - type: llm-rubric + value: > + The response must describe the Must Fix table as having a "Fixed" + column (Yes/No), since auto-fix applies only to must_fix issues. The + Should Fix and Suggestions tables must NOT include a Fixed column — + they only have ID, Issue, File, Confidence, and Suggested Fix. + # ================================================================== # tidy — all flag # ================================================================== From c1bf3b576c432317723d20b100c17cdcfbf6b57f Mon Sep 17 00:00:00 2001 From: Will Pike <6687499+pike00@users.noreply.github.com> Date: Fri, 8 May 2026 12:40:49 -0500 Subject: [PATCH 4/6] Emphasize global ID rules in again skill; add format regression tests Make Phase 2's "no restart" wording explicit and pull the user-reference syntax ("apply fix 1, 3, 5") out of a parenthetical into its own step, and call out the three-table layout in the Phase 3 intro. Add a test_again_numbering_format suite to scripts/test.sh that greps for the numbering, Suggested Fix column, '#' ID column, full-text aggregate rule, and pipe-escape rule so future edits cannot silently drop them. --- scripts/test.sh | 76 +++++++++++++++++++++++++++++++++++++++ src/skills/again/SKILL.md | 5 +-- 2 files changed, 79 insertions(+), 2 deletions(-) diff --git a/scripts/test.sh b/scripts/test.sh index 6eadb97..52e1c86 100755 --- a/scripts/test.sh +++ b/scripts/test.sh @@ -177,6 +177,78 @@ test_argument_handling() { done } +test_again_numbering_format() { + # Verify the again skill documents the global-ID + Suggested Fix format + # added on feat/numbered-suggested-fixes. These checks catch regressions + # if a future edit removes the numbering scheme or the Suggested Fix + # column from the summary template. + + local skill="$PROJECT_ROOT/src/skills/again/SKILL.md" + local relpath="src/skills/again/SKILL.md" + + if [[ ! -f "$skill" ]]; then + fail "$relpath not found" + return + fi + + local body + body=$(cat "$skill") + + # Phase 2 must instruct global sequential numbering. + if echo "$body" | grep -qE "global sequential ID"; then + pass "$relpath: documents global sequential ID assignment" + else + fail "$relpath: missing 'global sequential ID' instruction in Phase 2" + fi + + # Phase 2 must explain that IDs do not restart between severity tables. + if echo "$body" | grep -qE "numbering does not restart"; then + pass "$relpath: documents that numbering does not restart between tables" + else + fail "$relpath: missing 'numbering does not restart' clarification" + fi + + # Phase 2 must store the assigned number on each issue as 'id'. + if echo "$body" | grep -qE "\`id\` field"; then + pass "$relpath: documents 'id' field on aggregated issues" + else + fail "$relpath: missing 'id' field documentation in Phase 2" + fi + + # Each summary table must include a 'Suggested Fix' column. + local sf_count + sf_count=$(grep -c "Suggested Fix" "$skill" || true) + if [[ "$sf_count" -ge 3 ]]; then + pass "$relpath: 'Suggested Fix' column present in all 3 summary tables" + else + fail "$relpath: 'Suggested Fix' column missing (found $sf_count occurrences, need >= 3)" + fi + + # Each summary table must include a '#' column for the global ID. + # The header row is '| # | Issue | ...'; require at least 3 such headers. + local id_col_count + id_col_count=$(grep -cE '^\| #[[:space:]]*\| Issue' "$skill" || true) + if [[ "$id_col_count" -ge 3 ]]; then + pass "$relpath: '#' ID column present in all 3 summary tables" + else + fail "$relpath: '#' ID column missing (found $id_col_count, need >= 3)" + fi + + # Truncation rule: conversation may collapse, but full text lives in aggregate.md. + if echo "$body" | grep -qE "full text always lives in"; then + pass "$relpath: documents that full Suggested Fix text lives in aggregate.md" + else + fail "$relpath: missing 'full text always lives in aggregate.md' rule" + fi + + # Pipe/newline escaping rule for the Suggested Fix cell. + if echo "$body" | grep -qE "escape pipe characters"; then + pass "$relpath: documents pipe-escaping rule for Suggested Fix cell" + else + fail "$relpath: missing pipe-escaping rule for Suggested Fix cell" + fi +} + test_cross_references() { local pjson="$PROJECT_ROOT/src/dot-claude-plugin/plugin.json" @@ -366,6 +438,10 @@ echo "--- argument handling ---" test_argument_handling echo "" +echo "--- again numbering + Suggested Fix format ---" +test_again_numbering_format +echo "" + echo "--- cross-references ---" test_cross_references echo "" diff --git a/src/skills/again/SKILL.md b/src/skills/again/SKILL.md index 3b7e8cb..c17e7b2 100644 --- a/src/skills/again/SKILL.md +++ b/src/skills/again/SKILL.md @@ -88,7 +88,8 @@ After completing the configured number of passes, if `must_fix` issues remain an 1. **Deduplicate** on (file, title). Same issue across passes = higher confidence. 2. **Score**: Confidence = (passes finding issue) / (total passes) x 100%. 3. **Group** by severity, sort by confidence within groups. -4. **Number**: Assign a single global sequential ID (`1..N`) across all issues. Order: every `must_fix` first (in confidence-sorted order), then every `should_fix`, then every `suggestion`. The same ID identifies the issue in the conversation tables, in `aggregate.md`, and in `aggregate.json` (store as the `id` field on each issue). The user can reference fixes by these numbers (e.g. "apply fix 1, 3, 5"). +4. **Number**: Assign a single global sequential ID (`1..N`) across all issues. Order: every `must_fix` first (in confidence-sorted order), then every `should_fix`, then every `suggestion`. **Numbering is global and does NOT restart between the three severity tables** - if `must_fix` ends at ID k, `should_fix` starts at k+1, and `suggestions` continues from there. The same ID identifies the issue in the conversation tables, in `aggregate.md`, and in `aggregate.json` (store as the `id` field on each issue). +5. **Reference by ID**: Because IDs are unique and global, the user can refer to specific fixes by number (e.g. "apply fix 1, 3, 5") to selectively apply or skip individual issues across any of the three tables. Tell the user this in the final summary. ## Phase 3: Save and Report @@ -98,7 +99,7 @@ Save to `.lookagain//`: `aggregate.md` uses the same tables shown below (same columns, same global IDs). Keep `suggested_fix` text intact in the markdown file - do not truncate. To preserve the table format, escape pipe characters as `\|` and replace embedded newlines with `
` (or single spaces) inside the `Suggested Fix` cell. -Present the final summary to the user in this format: +Present the final summary to the user. The summary contains **three separate tables**, one for each severity (Must Fix, Should Fix, Suggestions), each with a `Suggested Fix` column and the global `#` ID column. Use this exact format: ``` ## Iterative Review Complete From 7edc68b987d79126699fbcb385239108b01d43a7 Mon Sep 17 00:00:00 2001 From: Will Pike <6687499+pike00@users.noreply.github.com> Date: Fri, 8 May 2026 15:54:11 -0500 Subject: [PATCH 5/6] Bump eval max_tokens to 4096 Plans for the again skill now exceed 2048 tokens because Phase 2 was expanded with explicit "no restart" wording and a separate Reference-by-ID step. Tests 8-10 were truncating mid-sentence before the summary-format section and failing rubric checks. 4096 leaves headroom for the full plan. --- evals/promptfooconfig.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/evals/promptfooconfig.yaml b/evals/promptfooconfig.yaml index 1c5af1b..05952a4 100644 --- a/evals/promptfooconfig.yaml +++ b/evals/promptfooconfig.yaml @@ -6,7 +6,7 @@ prompts: providers: - id: anthropic:messages:claude-sonnet-4-6 config: - max_tokens: 2048 + max_tokens: 4096 tests: # ================================================================== From befc11b5ae9bef2cc6c34531750668f718bdf585 Mon Sep 17 00:00:00 2001 From: Will Pike <6687499+pike00@users.noreply.github.com> Date: Fri, 8 May 2026 15:55:23 -0500 Subject: [PATCH 6/6] Add .env to .gitignore to exclude environment configuration files --- .gitignore | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.gitignore b/.gitignore index 18b3190..61b4ab5 100644 --- a/.gitignore +++ b/.gitignore @@ -21,3 +21,5 @@ Thumbs.db # Temporary files *.tmp *.temp + +.env \ No newline at end of file