Skip to content

feat(review-learnings): explicit @bot remember + autonomous capture (#160) - #162

Merged
chrisleekr merged 3 commits into
mainfrom
feat/learning-capture
May 20, 2026
Merged

feat(review-learnings): explicit @bot remember + autonomous capture (#160)#162
chrisleekr merged 3 commits into
mainfrom
feat/learning-capture

Conversation

@chrisleekr

@chrisleekr chrisleekr commented May 20, 2026

Copy link
Copy Markdown
Owner

Closes #160. Implements Option A + Option C per the maintainer's clarifications:

  • Option A can fire from issue, PR, or PR-review comment anywhere; the handler walks the discussion-digest so @chrisleekr-bot remember this on its own resolves against upstream thread context.
  • Option C captures autonomously, no propose step; the agent itself decides whether a maintainer rebuttal (resolve) or a digest directive (review) is durable policy worth persisting.

Architecture

flowchart TD
    UserA["@bot remember &lt;rule&gt;<br/>(any comment)"]:::ext --> Intent["intent-classifier<br/>maps to 'remember'"]
    Intent --> Reg["registry: bot:remember<br/>context=both"]
    Reg --> Handler["remember.ts handler<br/>narrowed tool surface"]
    Handler --> Digest["discussion-digest<br/>(thread context)"]
    Handler --> Pipe["runPipeline<br/>enableReviewLearnings=true"]
    Pipe --> Agent["Claude Agent<br/>extract + dedup + save"]
    Agent -->|save_review_learning| Sink[("review_learnings table<br/>via orchestrator drain")]:::data

    UserB["maintainer rebuts<br/>bot finding"]:::ext --> Resolve["resolve.ts agent<br/>classifies Invalid"]
    Resolve -->|autonomous decision| Agent
    UserC["digest directive<br/>contradicts a finding"]:::ext --> Review["review.ts agent<br/>cross-check"]
    Review -->|autonomous decision| Agent

    classDef ext fill:#2c3e50,color:#ffffff
    classDef data fill:#27ae60,color:#ffffff
Loading

Option C: autonomous save in resolve.ts + review.ts

Prompt-only changes (no infra, no DB, no MCP). Both handlers already enable review_learnings via enableReviewLearnings: true (PR #161); this change tightens the agent's instructions:

  • resolve.ts buildResolvePrompt extends step 5's Invalid branch with an explicit autonomous-save sub-instruction. The agent decides for itself whether the rebuttal's rationale represents durable repo policy and calls mcp__repo_memory__save_review_learning directly. Skips one-off rationales ("we'll remove this next week"); persists durable ones with full provenance.
  • review.ts buildReviewPrompt extends step 8 (issue-class list) with a cross-check before posting any finding. When a digest's maintainer-authoritative directive would suppress the finding AND that directive isn't already in the <review_learnings_…> block, the agent saves it for future runs that lack the exact digest context.

Option A: bot:remember workflow

A new registered workflow handles explicit directive capture:

Surface Change
src/workflows/registry.ts 7th entry: remember, label bot:remember, context both, no prior, no steps. WorkflowNameSchema widened.
src/workflows/intent-classifier.ts IntentWorkflowSchema + system prompt teach the classifier to route comments starting with "remember" (post bot mention) to the new workflow, with guardrails against picking remember for free-form "remember to do X" tasks.
src/workflows/handlers/remember.ts (new) Fetches the digest, seeds a tracking comment, runs runPipeline with a narrowed tool surface (save_review_learning, get_review_learnings, update_claude_comment only, no Bash, no Edit, no commits). Prompt covers both trigger forms, enforces a get_review_learnings dedup pre-check, refuses cleanly when no directive can be extracted, and writes a REMEMBER.md audit log that becomes the tracking-comment body.
src/orchestrator/job-queue.ts Workflow-name enum lists remember. Hardcoded rather than imported from registry.ts to avoid the job-queue → registry → handlers/ship → job-queue cycle.
Docs New docs/use/workflows/remember.md. Updated docs/use/workflows/index.md (seven workflows). Expanded docs/use/review-learnings.md with the three-path capture model. mkdocs nav gains the new page.
Tests Registry tests cover the new workflow + intent. Intent-classifier fixture set gains 3 remember cases.

Trigger forms

Inline -- directive sits in the trigger comment:

@chrisleekr-bot remember: do not flag fixture duplication in test/**/*.test.ts

Referential -- directive lives upstream in the thread:

> [earlier maintainer comment]
> We keep these per-file rather than centralising; the closure needs
> the literal at module-evaluation time.

@chrisleekr-bot remember this

The handler always feeds the agent the full discussion digest, so the referential form has the context it needs.

Refusal model

The agent doesn't save (and says so on the tracking comment) when:

  • Trigger is referential but the upstream thread carries no policy-shaped maintainer statement.
  • Proposed directive paraphrases an existing entry (agent dedupes via get_review_learnings).
  • Directive collapses to empty after sanitization.

Static gates (local)

Gate Result
bun run typecheck 0 errors
bun run lint 0 errors, 414 warnings (baseline-aligned)
bun run format clean
bun run scripts/em-dash-sweep.ts --check zero in scope
bun run docs:build (strict) clean
bun run scripts/check-docs-versions.ts clean
bun run scripts/check-docs-citations.ts clean
bun run scripts/check-action-pins.ts clean
bun run scripts/check-docs-sync.ts passes (workflow files + docs co-touched)
bun test test/workflows/registry.test.ts test/workflows/intent-classifier.test.ts 22/22 pass

DB-dependent suites skipped (TEST_DATABASE_URL not set locally).

Follow-ups deferred to separate PRs

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Introduced @bot remember command to capture and persist review policy statements as durable repository knowledge.
  • Improvements

    • Enhanced system to autonomously capture policy learnings during standard review and resolution workflows.
  • Documentation

    • Updated documentation describing the multi-path learning capture model and new remember workflow functionality.

Review Change Stack

…160)

Closes #160. Implements Option A (explicit `@chrisleekr-bot remember`)
and Option C (autonomous capture in review/resolve) per the maintainer's
clarifications:

* Option A can fire from issue, PR, or PR-review comment anywhere; the
  handler walks the discussion-digest so `@chrisleekr-bot remember this`
  on its own resolves against upstream thread context.
* Option C captures autonomously, no propose step; the agent itself
  decides whether a maintainer rebuttal (resolve) or a digest directive
  (review) is durable policy worth persisting.

## Option C: autonomous save in resolve.ts + review.ts

Prompt-only changes (no infra, no DB, no MCP, no tests beyond the
prompt strings):

- `resolve.ts buildResolvePrompt`: extends step 5's `Invalid`
  classification with an explicit "decide autonomously and call
  `mcp__repo_memory__save_review_learning`" sub-instruction. Skips
  one-off rationales ("we'll remove this next week"); persists durable
  ones with full provenance and no propose / confirmation step.
- `review.ts buildReviewPrompt`: extends step 8 (issue-class list) with
  a cross-check, before posting a finding, against (a) the existing
  `<review_learnings_…>` block and (b) the digest's maintainer-
  authoritative directives. When a digest directive suppresses a
  finding AND is not already in the block, the agent calls
  save_review_learning to persist it for future runs that lack the
  exact digest context.

The save round-trip uses the existing repo_memory MCP tool / daemon-
actions / orchestrator persistence path landed in PR #161; nothing new
on the data plane.

## Option A: `bot:remember` workflow

A new registered workflow handles explicit directive capture:

- `WorkflowNameSchema` and the registry gain a 7th entry: `remember`,
  label `bot:remember`, context `both`, no prior, no steps.
- `IntentWorkflowSchema` and the classifier system prompt teach the
  classifier to route comments starting with "remember" (post bot
  mention) to the new workflow, with explicit guardrails against
  picking remember for free-form "remember to do X" tasks.
- `src/workflows/handlers/remember.ts` (new): fetches the discussion
  digest, seeds a tracking comment, runs `runPipeline` with a narrowed
  tool surface (`save_review_learning`, `get_review_learnings`,
  `update_claude_comment` only — no Bash, no Edit, no commits). The
  prompt covers both trigger forms (inline `remember: <rule>` and
  referential `remember this` / `remember the rule above`), enforces a
  `get_review_learnings` dedup pre-check, refuses cleanly when no
  directive can be extracted, and writes a REMEMBER.md audit log that
  becomes the tracking-comment body.

Wire details:

- `src/orchestrator/job-queue.ts` workflow-name enum now lists
  `remember`. Hardcoded rather than imported from registry.ts to avoid
  the `job-queue → registry → handlers/ship → job-queue` cycle; the
  duplicate is small and surfaces via type-error at every enqueueJob
  call site if drift creeps in.
- Docs: new `docs/use/workflows/remember.md`, updated
  `docs/use/workflows/index.md` (seven workflows now), expanded
  `docs/use/review-learnings.md` with the three-path capture model.
- mkdocs nav gains the new page; fixtures + registry tests cover the
  new workflow + intent.

## Static gates

typecheck, lint (0 errors), format, em-dash sweep, docs build strict,
version pins, citation checks, action pins, docs-sync, unit tests for
the touched surface (22/22). DB-dependent suites skipped per usual.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings May 20, 2026 14:17
@coderabbitai

coderabbitai Bot commented May 20, 2026

Copy link
Copy Markdown

Warning

Rate limit exceeded

@chrisleekr has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 46 minutes and 2 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: b236fdb7-da03-4138-9ee8-6df68359bbc9

📥 Commits

Reviewing files that changed from the base of the PR and between 3cb665d and 6a2803d.

📒 Files selected for processing (2)
  • docs/use/workflows/index.md
  • docs/use/workflows/remember.md
📝 Walkthrough

Walkthrough

This PR implements the remember workflow, enabling explicit @bot remember commands to capture maintainer-authored review-policy directives into persistent review_learnings. The workflow integrates with the existing intent classifier, runs a constrained pipeline with repo-memory MCP tools, and supplements autonomous capture paths in resolve and review handlers. Full documentation and registry tests accompany the feature.

Changes

Remember Workflow for Explicit Policy-Learning Capture

Layer / File(s) Summary
Workflow registration and intent routing
src/workflows/registry.ts, src/orchestrator/job-queue.ts, src/workflows/intent-classifier.ts
remember is added to WorkflowNameSchema, job-queue schema, and intent-classifier system prompt so the workflow is discoverable and recognizable from comment text.
Pipeline integration for unfiltered learnings
src/core/pipeline.ts
New unfilteredReviewLearnings?: boolean override in RunPipelineOverrides allows remember to bypass file-applicability filtering and access the full learnings set during dedup pre-checks.
Remember handler implementation
src/workflows/handlers/remember.ts
Handler processes @bot remember [...] directives by fetching target metadata, building discussion context, constructing agent instructions, and running a constrained pipeline with repo-memory save/dedup tools; includes fetchTitleAndBody helper and buildRememberPrompt for LLM instruction generation.
Autonomous capture integration in resolve and review
src/workflows/handlers/resolve.ts, src/workflows/handlers/review.ts
Prompt updates instruct agents to autonomously save policy learnings: resolve saves maintainer rationale when Invalid rebuttals reflect durable policy; review persists digest directives as learnings when they suppress findings not already captured.
Documentation for remember workflow and review learnings
docs/use/workflows/remember.md, docs/use/review-learnings.md, docs/use/workflows/index.md, mkdocs.yml
New remember workflow documentation covers directive formats (inline vs referential), capture paths, refusal conditions, trust boundaries, and scope/provenance handling; updates to learnings and workflows overview docs reflect the multi-path capture model and seven-workflow registry.
Registry tests and intent fixture updates
test/workflows/registry.test.ts, test/workflows/fixtures/intent-comments.json
Registry tests updated to validate seven canonical workflow names including remember; intent-comments fixture expanded with new remember directive examples (inline rules and referential anchors) and existing workflow examples normalized for consistency.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

Suggested labels

type: docs 📋, feature: review-learnings 🧠

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main changes: adding explicit @bot remember workflow and autonomous policy capture by agents during review/resolve.
Linked Issues check ✅ Passed The PR implements Options A and C from issue #160: explicit @bot remember workflow [#160] and autonomous capture in resolve/review handlers [#160].
Out of Scope Changes check ✅ Passed All changes directly support the two implementation options: remember handler, intent classifier updates, workflow registry, documentation, and prompt modifications for autonomous capture.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an explicit bot:remember workflow for maintainers to persist durable review-policy directives, and tightens review/resolve prompts to autonomously capture maintainer directives as review_learnings for future runs.

Changes:

  • Registers a new remember workflow (bot:remember) and teaches the intent classifier to route @… remember … triggers.
  • Introduces a new remember handler that runs a narrowed pipeline to extract + dedup + persist a review-learning directive from thread context.
  • Updates docs/mkdocs navigation and tests/fixtures to cover the new workflow and capture paths.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
test/workflows/registry.test.ts Updates registry and schema tests to include the new remember workflow.
test/workflows/fixtures/intent-comments.json Adds classifier fixtures for remember triggers (inline + referential).
src/workflows/registry.ts Registers the remember workflow and widens WorkflowNameSchema.
src/workflows/intent-classifier.ts Extends classifier system prompt with remember routing guardrails.
src/workflows/handlers/review.ts Adds prompt instruction for autonomous persistence of digest directives as learnings.
src/workflows/handlers/resolve.ts Adds prompt instruction for autonomous persistence on Invalid rebuttals.
src/workflows/handlers/remember.ts New handler to extract/dedup/save review-learnings from explicit remember invocations.
src/orchestrator/job-queue.ts Extends workflow-name enum to include remember.
mkdocs.yml Adds bot:remember page to docs navigation.
docs/use/workflows/remember.md Documents the new bot:remember workflow behavior and trust boundary.
docs/use/workflows/index.md Updates workflow index from six to seven workflows.
docs/use/review-learnings.md Documents the three capture paths, including explicit remember.

Comment thread src/workflows/handlers/remember.ts
Comment thread src/workflows/handlers/remember.ts
Comment thread src/workflows/handlers/remember.ts
Comment thread src/workflows/handlers/remember.ts Outdated
Comment thread src/workflows/handlers/remember.ts Outdated
Five threads on the remember workflow, all classified Valid:

1. **handlers/remember.ts:126** — `enableReviewLearnings: true` causes
   `runPipeline` to filter `ctx.reviewLearnings` to PR-changed-file
   matches before the MCP env sees it. For `bot:remember` that broke
   the dedup pre-check: directives outside the current PR's diff
   (or every glob-scoped directive on issue-context triggers) would
   be invisible to `get_review_learnings`. Added an
   `unfilteredReviewLearnings` opt-in to `RunPipelineOverrides`; the
   pipeline skips the applicability filter when set. The remember
   handler passes it, so the agent's dedup sees the full universe.

2. **handlers/remember.ts:122** — The narrowed `allowedTools` list
   omitted `Write`, but the prompt required the agent to write
   `$BOT_ARTIFACT_DIR/REMEMBER.md`. The file was never created and
   the tracking-comment finalisation always fell back to the
   "no audit produced" message. Resolved by dropping the file
   captureFile entirely; the audit log is now posted directly to
   the tracking comment via `update_claude_comment`, which the
   workflow already exposes. The artifact row in the docs and the
   handler's success path are updated accordingly.

3. **handlers/remember.ts:127** — `enableGithubState` defaults to
   `true` on PR targets in `pipeline.ts`, so the github-state MCP
   tools were silently appended to my narrowed allowlist on PR-
   triggered remembers. Passed `enableGithubState: false` to keep
   the surface as advertised in the prompt.

4. **handlers/remember.ts:245** — The prompt's `file_glob` guidance
   claimed brace alternations like `test/**/*.{test,spec}.ts` are
   "pathological". They're not: `isSafeGlob` accepts up to 6 groups
   and 8 alternates. Updated the prompt to match the real
   validation rules so the agent does not unnecessarily drop valid
   precise globs.

5. **handlers/remember.ts:237** — Step 2 required the trigger
   comment itself to be from a maintainer, but the workflow is
   routed purely by intent classification and the docs explicitly
   allow non-owners to point at an upstream maintainer rule via
   "remember this". Step 2 now treats the trigger as
   author-agnostic; step 3 still requires the directive *source* to
   be maintainer-authoritative (the discussion digest's owner-
   directives section is the trust boundary, unchanged).

Local gates: typecheck, lint (0 errors, baseline warnings), format,
em-dash sweep, docs build strict, relevant unit tests (22/22 pass).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/use/workflows/index.md`:
- Line 13: Update the artifact column for the `remember` row in the workflows
doc to indicate no file artifact is produced: replace `REMEMBER.md` with `N/A`
(or similar). The `remember` handler (see remember handler in
src/workflows/handlers/remember.ts) uses `save_review_learning` and
`update_claude_comment` and explicitly avoids writing a capture file, so the
documentation should reflect that no file is written. Ensure the table entry for
[`remember`](remember.md) / `bot:remember` is updated accordingly.

In `@docs/use/workflows/remember.md`:
- Around line 35-47: Two fenced code blocks containing the plain text directives
(the block starting with "`@chrisleekr-bot` remember: do not flag fixture
duplication in test/**/*.test.ts" and the block containing the quoted maintainer
comment and "`@chrisleekr-bot` remember this") are unlabeled and trigger
markdownlint MD040; update each triple-backtick fence to include a language
identifier (e.g., add "text" after the opening ```), so the fences become
```text for both the single-line directive block and the quoted multi-line
block, preserving their exact content and indentation.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: a719cdb5-7d87-435d-948d-f18f653158bf

📥 Commits

Reviewing files that changed from the base of the PR and between ba50972 and 3cb665d.

📒 Files selected for processing (13)
  • docs/use/review-learnings.md
  • docs/use/workflows/index.md
  • docs/use/workflows/remember.md
  • mkdocs.yml
  • src/core/pipeline.ts
  • src/orchestrator/job-queue.ts
  • src/workflows/handlers/remember.ts
  • src/workflows/handlers/resolve.ts
  • src/workflows/handlers/review.ts
  • src/workflows/intent-classifier.ts
  • src/workflows/registry.ts
  • test/workflows/fixtures/intent-comments.json
  • test/workflows/registry.test.ts

Comment thread docs/use/workflows/index.md Outdated
Comment thread docs/use/workflows/remember.md Outdated
…uages

Two doc nits from CodeRabbit on PR #162 after the remember handler
dropped its `captureFiles`:

- `docs/use/workflows/index.md` catalog row still listed `REMEMBER.md`
  for `bot:remember`; updated to `tracking comment` so the catalog
  matches the per-page artifact field already corrected in 3cb665d.
- `docs/use/workflows/remember.md` lines 35 + 41: fenced code blocks
  carried no language identifier (markdownlint MD040). Added `text`.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@chrisleekr
chrisleekr merged commit 1c4c53a into main May 20, 2026
22 checks passed
@chrisleekr
chrisleekr deleted the feat/learning-capture branch May 20, 2026 15:05
chrisleekr pushed a commit that referenced this pull request May 21, 2026
# [1.13.0](v1.12.2...v1.13.0) (2026-05-21)

### Bug Fixes

* **deps:** update dependency @anthropic-ai/bedrock-sdk to ^0.29.0 ([#147](#147)) ([eb95c64](eb95c64))
* **deps:** update dependency @anthropic-ai/claude-agent-sdk to ^0.3.0 ([#154](#154)) ([15add8e](15add8e))
* **docs:** anchor-verify src citations to catch silent line-shift rot ([#163](#163)) ([5a67863](5a67863))
* **webhook:** subscribe issue_comment.edited/.deleted for cache write-through ([#131](#131)) ([c84361d](c84361d))
* **webhook:** write-through target_cache on issues/pull_request events ([#130](#130)) ([#132](#132)) ([8b79c10](8b79c10))

### Features

* **prompt:** opt-in cacheable system/user prompt split ([#135](#135)) ([bb80ca7](bb80ca7))
* **review-learnings:** explicit [@bot](https://github.com/bot) remember + autonomous capture ([#160](#160)) ([#162](#162)) ([1c4c53a](1c4c53a))
* **review-learnings:** persistent per-repo review-policy directives ([#161](#161)) ([ba50972](ba50972))
* **scheduler:** scheduled actions via .github-app.yaml ([#159](#159)) ([142a5bc](142a5bc))
* **workflows:** comment-aware structured workflows via LLM discussion digest ([#148](#148)) ([7a6b315](7a6b315))
@chrisleekr

Copy link
Copy Markdown
Owner Author

🎉 This PR is included in version 1.13.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(review-learnings): implicit capture of maintainer directives from natural-language comments

2 participants