diff --git a/README.ja.md b/README.ja.md index cad4b08..583c621 100644 --- a/README.ja.md +++ b/README.ja.md @@ -95,7 +95,15 @@ GitHub webhooks + GitHub API ## MCP Tools -この MCP サーバーが公開するツールは 1 つに統合されています。意味検索、時系列 activity scan、doc 本文取得、`vector_id` 指定の保存済み本文取得のいずれも `search` のパラメータ経由で扱えます。以前の build で分かれていた `get_issue_context` / `get_doc_content` / `list_recent_activity` は削除され、用途は下記パラメータに吸収されました。 +この MCP サーバーは `search` と3つの非公開 feedback ツールを公開します。意味検索、時系列 activity scan、doc 本文取得、`vector_id` 指定の保存済み本文取得のいずれも `search` のパラメータ経由で扱えます。以前の build で分かれていた `get_issue_context` / `get_doc_content` / `list_recent_activity` は削除され、用途は下記パラメータに吸収されました。 + +### 非公開 feedback ツール + +成功検索は既定で UTC trace を保存し、資料の版を指す `source_id` と現在 activation を返します。`memory_history` で履歴・利用段階・取消し audit を読み、`record_source_use` で selected→validated→used、`record_outcome` で confirmed または receipt を指定した corrected/rolled_back を記録します。検索だけでは used/confirmed になりません。保存不可の結果には trace がなく feedback 不可です。半減期3600秒、activation は各 channel 上限10（total20）、graph strength 上限5です。成功検索に DO request と保存コストが加わります。 + +live inline doc/wiki 本文は返した本文の SHA-256 で版を識別し、`github_live` provenance を持ちます。stored fetch と graph 本文は index snapshot の版で、索引 timestamp が同じでも live 本文の変更は別 source_id になります。 + +[仕様・移行・合成実演](docs/2-feedback-memory.ja.md) を参照してください。0008 を Worker deploy より先に適用し、新 schema JSON を bridge artifact に同梱します。検索品質向上は未評価です。 ### `search` @@ -134,8 +142,9 @@ bot (`sender.login` が `[bot]` で終わる) と trim 後 10 文字未満の bo | `until` | ISO 8601 文字列 | `updated_at < until` の結果だけを残します。 | | `include_content` | boolean | 上位 doc 結果に本文を inline する (既定 `false`)。 | | `vector_ids` | string[] | 保存済み本文の取得。読み出す行の `vector_id`、1 回あたり最大 50 件。他モードより優先され、指定時は `query` / `sort` と全 filter が無視されます。下記「保存済み本文の取得」参照。 | -| `graph_expand` | boolean | opt-in の GraphRAG 拡張（search モードのみ）。`true` のとき fusion 後の上位結果を seed に Decision-Structure の mention グラフ（D1 `doc_edges`）を辿り、関連 wiki ページを `graph_hop` / `graph_from` 付きで別配列 `graph_results` として返す（下記「検索の 2 軸」参照）。既定 `false` は標準ハイブリッド検索とバイト単位で同一（グラフ未参照）。 | +| `graph_expand` | boolean | opt-in の GraphRAG 拡張（search モードのみ）。`true` のとき fusion 後の上位結果を seed に Decision-Structure の mention グラフ（D1 `doc_edges`）を辿り、関連 wiki ページを `graph_hop` / `graph_from` 付きで別配列 `graph_results` として返す（下記「検索の 2 軸」参照）。既定 `false` は標準ハイブリッドの順位を保つ（グラフ未参照）。trace metadata は追加される。 | | `graph_hops` | number | `graph_expand` のグラフ探索深度（1 または 2、既定 1）。`graph_expand` が `false` のときは無視。 | +| `use_memory` | boolean | 既定 false。true は最大200 graph 候補を同 hop 内で learned strength 順にしてから返却上限へ絞る。keyword の score/rank は維持。 | #### `type` 値 diff --git a/README.md b/README.md index 3bbcf7a..1b242af 100644 --- a/README.md +++ b/README.md @@ -95,7 +95,15 @@ See: ## MCP Tools -This MCP server exposes a single consolidated tool. All retrieval modes — semantic search, time-ordered activity scan, inline doc content fetch, and stored-content fetch by `vector_id` — are reached through `search` via its parameter set. Earlier builds split these across `get_issue_context`, `get_doc_content`, and `list_recent_activity`; those tools have been removed and their use cases now fold into the parameters below. +This MCP server exposes `search` and three private feedback tools. All retrieval modes — semantic search, time-ordered activity scan, inline doc content fetch, and stored-content fetch by `vector_id` — are reached through `search` via its parameter set. Earlier builds split these across `get_issue_context`, `get_doc_content`, and `list_recent_activity`; those tools have been removed and their use cases now fold into the parameters below. + +### Private feedback tools + +Successful retrieval writes a UTC trace by default, returning version-specific `source_id` and current activation. `memory_history` reads history, stages and reversal audit; `record_source_use` records selected→validated→used; `record_outcome` confirms a saved path or corrects/rolls back one named receipt. Retrieval alone is never usage or confirmation. Memory failure emits no usable trace. Half-life is 3600 seconds, each activation channel caps at 10 (total 20), and relation strength caps at 5. Successful retrieval adds a DO request and storage cost. + +Live inline doc/wiki bodies use the returned-text SHA-256 version and `github_live` provenance. Stored fetch and graph bodies refer to the index snapshot; changed live text has another source ID even with the same index timestamp. + +See the [specification, migration and synthetic lifecycle](docs/2-feedback-memory.md). Apply 0008 before Worker deployment and include the new schema JSON in bridge artifacts. Retrieval quality improvement has not been evaluated. ### `search` @@ -134,8 +142,9 @@ Bot-authored comments (`sender.login` ending in `[bot]`) and comments shorter th | `until` | ISO 8601 string | Keep only results with `updated_at < until`. | | `include_content` | boolean | Inline raw content on top doc results (default `false`). | | `vector_ids` | string[] | Stored-content fetch. The `vector_id` values of the rows to read back, max 50 per call. Takes precedence over the other modes: `query`, `sort` and every filter are ignored when present. See [Stored-content fetch](#stored-content-fetch). | -| `graph_expand` | boolean | Opt-in GraphRAG expansion (search mode only). When `true`, after fusion the top results seed a traversal of the Decision-Structure mention graph (D1 `doc_edges`); related wiki pages come back in a separate `graph_results` array tagged `graph_hop` / `graph_from` — see Retrieval axes below. Default `false` = byte-identical to standard hybrid retrieval (no graph read). | +| `graph_expand` | boolean | Opt-in GraphRAG expansion (search mode only). When `true`, after fusion the top results seed a traversal of the Decision-Structure mention graph (D1 `doc_edges`); related wiki pages come back in a separate `graph_results` array tagged `graph_hop` / `graph_from` — see Retrieval axes below. Default `false` preserves standard hybrid ranking (no graph read); trace metadata is added. | | `graph_hops` | number | Graph traversal depth for `graph_expand` (1 or 2, default 1). Ignored when `graph_expand` is `false`. | +| `use_memory` | boolean | Default false. True orders up to 200 graph candidates by learned strength within equal hops before the output cap. Keyword scores/ranks are preserved. | #### `type` values diff --git a/docs/0-requirements.ja.md b/docs/0-requirements.ja.md index 71d20f8..c414126 100644 --- a/docs/0-requirements.ja.md +++ b/docs/0-requirements.ja.md @@ -83,16 +83,7 @@ Responsibilities: - semantic retrieval と structured retrieval の tool を公開する - downstream agent がそのまま使える形式で state を返す -スキーマの source of truth は Worker 側の tool 定義。client proxy -(`mcp-server/server/tools.js`) は `tools/list` を Worker へ転送せず、`search` -params の静的ミラーから応答する（起動時を auth/network なしに保つため）。ミラーは -手動保守ゆえ、Worker に param を足して proxy を忘れると、MCP クライアントが -`additionalProperties: false` で黙って落とし Worker に届かない（gh#157）。 -`scripts/check-schema-drift.mjs` を CI で実行し、proxy の `search` スキーマが Worker -からズレたら build を失敗させる。両者は同期せずに出荷できない。 -比較する軸は 2 つ、param 名と enum param の値。名前だけの比較では proxy の `type` -enum から `wiki_doc` が抜けたまま気付けず、Worker は受け付けるのにクライアント側で -値が弾かれていた（gh#181）。 +Worker の Zod contract を source of truth とし、bridge は生成した静的 JSON で `tools/list` を返す。起動時の auth/network は不要。`scripts/generate-tool-contracts.mjs` で再生成し、`scripts/check-schema-drift.mjs` は全4 tool の実 protocol と nested input/output schema、bounds、defaults、annotations の一致を CI で検査する。名前/enum のみだった旧 guard を拡張する（#259）。 #### プロトコル版 2026-07-28、単レーン @@ -117,7 +108,7 @@ README 注意書きで吸収する。 - かつて MCP を提供していた Durable Object（`RagMcpAgent` / `RagMcpAgentV2`）は 経路から外れた。過去の migration が名指しするクラスはスクリプト内に存在せねば ならないという Cloudflare の制約のため、退役スタブとして export だけ残す。撤去には - `deleted_classes` migration が要る。実データを保持する `IssueStore` は無変更。 + `deleted_classes` migration が要る。`IssueStore` は実データを保持し、#259 では独立 memory table を追加する。 この版は本 repository の 2 成果物のあいだの私的な契約である。Worker へ到達する経路は npx ブリッジ（`mcp-server/`）だけ——`server.json` は stdio トランスポートしか宣言 @@ -668,3 +659,11 @@ embedding が失敗した record は incomplete と分かる形で残し、次 - ranking と filtering の改善 - multi-agent handoff retrieval の改善 - cross-repository state recovery の改善 + +## 検索 feedback memory（Issue #259） + +検索履歴、selected/validated/used、半減期3600秒の retrieved/usage activation、confirmed-only の実 path 強化と receipt 指定取消しを同じ PR で実装する。認証済み GitHub principal ごとに共有・分離する。成功検索は trace を原子的に保存し、保存失敗や部分取得は trace を発行せず feedback 不可とする。検索は既定で履歴を書き、keyword の順位は維持する。use_memory は graph の同 hop 内だけに影響し、2 hop の実 edge を保存する。 + +新 API は `memory_history`、`record_source_use`、`record_outcome`。設定値、上限、ID/version、batch/idempotency、ledger、error、0008先行移行、既存行 backfill と artifact の契約は [2-feedback-memory.ja.md](2-feedback-memory.ja.md) に定義する。会話/本文/credential の複製は行わず、品質向上の主張は別評価を必要とする。 + +#259 の版契約: live inline doc/wiki は返した本文の SHA-256 を version とし、index snapshot と provenance を分ける。同じ索引 timestamp でも live 本文更新は別 source_id にする。本文を memory に複製しない。 diff --git a/docs/0-requirements.md b/docs/0-requirements.md index 18150be..0de8cd2 100644 --- a/docs/0-requirements.md +++ b/docs/0-requirements.md @@ -83,17 +83,7 @@ Responsibilities: - expose semantic and structured retrieval tools - return state in a format that downstream agents can consume directly -Schema source of truth: the Worker tool definitions are authoritative. The client -proxy (`mcp-server/server/tools.js`) answers `tools/list` from a static mirror of -the `search` params — it does not forward `tools/list` to the Worker, keeping -startup auth-free and network-free. Because that mirror is hand-maintained, a param -added to the Worker but forgotten in the proxy is silently dropped by MCP clients -(`additionalProperties: false`) and never reaches the Worker (gh#157). -`scripts/check-schema-drift.mjs` runs in CI and fails the build when the proxy -`search` schema drifts from the Worker, so the two cannot ship out of sync. -The check compares two axes: param names and, for enum params, their values — -a name-only comparison let the proxy `type` enum omit `wiki_doc` while the Worker -accepted it, so clients rejected the value before any request was sent (gh#181). +Worker Zod contracts are the source of truth. The bridge answers `tools/list` from generated static JSON without startup auth/network. `scripts/generate-tool-contracts.mjs` regenerates those artifacts; `scripts/check-schema-drift.mjs` compares all four tools against their actual served nested input/output schemas, bounds, defaults and annotations in CI. This extends the previous name/enum-only guard (#259). #### Protocol revision: 2026-07-28, single lane @@ -120,7 +110,7 @@ Consequences of statelessness: out of the serving path. Their classes remain exported as retired stubs because Cloudflare requires every class named in a past migration to exist in the script; removing them needs a `deleted_classes` migration. `IssueStore`, - which holds real data, is untouched. + which holds real data, gains independent memory tables in #259. The revision is a private contract between this repository's two artifacts. The npx bridge (`mcp-server/`) is the only way to reach the Worker — `server.json` @@ -674,3 +664,11 @@ If an embedding attempt fails, the state must remain detectable as incomplete so - stronger ranking and filtering behavior - better multi-agent handoff retrieval - better cross-repository state recovery + +## Retrieval feedback memory (Issue #259) + +Implement retrieval history, selected/validated/used records, retrieved/usage activation with a 3600-second half-life, and confirmed-only real-path reinforcement with receipt-based reversal atomically in one PR. Share and isolate by authenticated GitHub principal. Successful retrieval commits its trace atomically; memory/partial retrieval failures emit no trace and cannot accept feedback. History writes are enabled by default, keyword ranking is preserved, and use_memory changes only same-hop graph ordering. Save actual two-hop edges. + +The APIs are `memory_history`, `record_source_use`, and `record_outcome`. [2-feedback-memory.md](2-feedback-memory.md) owns constants, bounds, source version identity, batch/idempotency, ledger, errors, migration 0008-before-deployment, historical backfill and bridge artifacts. Do not duplicate conversations, bodies or credentials. Claims of retrieval quality improvement require separate evaluation. + +Issue #259 version contract: live inline doc/wiki uses the returned-text SHA-256 version and provenance separate from the index snapshot. Changed live text gets a different source ID even at the same index timestamp. Do not duplicate the body in memory. diff --git a/docs/1-memory-philosophy.ja.md b/docs/1-memory-philosophy.ja.md index 9494aad..c36fb89 100644 --- a/docs/1-memory-philosophy.ja.md +++ b/docs/1-memory-philosophy.ja.md @@ -76,3 +76,7 @@ session boundary は普通に起こる。 - source-of-truth GitHub artifact の代替 これはそれらの artifact から正しい state を引き戻すための retrieval layer である。 + +## 非公開 retrieval feedback + +[検索 feedback memory](2-feedback-memory.ja.md) は query、返却資料の provenance、明示的 usage/outcome を保存する。会話全体や本文を複製せず、GitHub artifact から recoverable state を取り出す立場を保つ。 diff --git a/docs/1-memory-philosophy.md b/docs/1-memory-philosophy.md index 00bbca3..19d863f 100644 --- a/docs/1-memory-philosophy.md +++ b/docs/1-memory-philosophy.md @@ -76,3 +76,7 @@ It is not: - a substitute for source-of-truth GitHub artifacts It is a retrieval layer that helps agents recover the right state from those artifacts. + +## Private retrieval feedback + +[Retrieval feedback memory](2-feedback-memory.md) records queries, returned source provenance and explicit usage/outcomes. It preserves the recoverable-state stance over GitHub artifacts without duplicating conversations or bodies. diff --git a/docs/2-feedback-memory.ja.md b/docs/2-feedback-memory.ja.md new file mode 100644 index 0000000..6df0c1c --- /dev/null +++ b/docs/2-feedback-memory.ja.md @@ -0,0 +1,85 @@ +# 非公開の検索 feedback memory + +言語: [English](2-feedback-memory.md) | 日本語 + +Issue #259 の4機能である検索履歴、資料の明示的利用、時間減衰する活性度、取り消せる関係強化を同じ trace 契約で提供する。保存対象は query、検索設定、返した資料の provenance と明示判断。会話全体、資料本文やタイトル、credential、認証 props は複製しない。 + +## 資料と trace の識別 + +成功した `search`（search / scan / stored-content fetch / 0件）は UTC trace を保存してから `trace_id`、`timestamp`、`feedback_available:true` を返す。資料と `same_entity.others` には `source_id`、`provenance`、現在の `activation` を追加する。request は query/filter/control、settings は mode、有効な scan 窓、fusion/rerank 実施結果と両軸を保存する。 + +provenance は `{repo,type,identity,version,content_source}`。identity は issue/PR number、release tag、doc/wiki path、diff の `[commit_sha,file_path]`、GitHub comment/review ID。`content_source:"index"` の version は取得した `updated_at`（doc/wiki は索引 snapshot の timestamp）。live inline doc/wiki 本文は `content_source:"github_live"` と `content_version:"sha256:<返した UTF-8 本文の hash>"` を持ち、この hash を provenance の version と source ID に使う。`index_updated_at` は索引 timestamp を補足 audit として保存し、live 版は定義しない。同じ live 本文は索引 timestamp が変わっても同じ版、本文が変われば索引 timestamp が固定でも別 source_id。stored-content fetch と graph 本文は索引コピーで、index provenance を維持する。memory に保存するのは hash/provenance のみで、live 本文は複製しない。mode、vector handle、任意の補足 field は ID の hash に含めない。 + +`source_id` は実体の特定版を識別する。同じ実体・同じ取得版は vector ID が移行しても同じ ID、更新版は別 ID になる。activation は版単位。mention strength は `[repo,src_slug,dst_slug,edge_kind]` 単位で、vector ID や資料版から独立している。reindex は principal 別の学習状態を消さず、元の mention topology を保持する。消えた mention は探索できず、過去 trace の path は監査と取消しのため残る。 + +principal は検証済み MCP OAuth props の numeric GitHub user ID からサーバーが決める。同じ GitHub ユーザーの client 間で共有し、caller 指定 identity は信頼しない。別ユーザーには `unknown_trace` / `unknown_confirmation` を返す。DO の内部 API は公開 route を持たない。 + +`memory_history {limit:10}` は新しい trace から一覧を返し、`next_cursor` で次ページへ進む。`memory_history {trace_id,limit:10}` は資料、利用段階、現在 activation、検索設定と古い順の receipt audit を返す。この詳細側 cursor は**同じ trace 内の receipt**用で、一覧側とは用途が異なる。cursor は principal と、詳細側では trace にも所属する。 + +limit は1..50、trace の資料 snapshot は300件、保存入力は300,000文字、query は4096文字、metadata string は256文字、usage batch は100 entry、reason/key は1000/128文字まで。SQL は owner/cursor index と有界 page を使う。全履歴をまとめて読む API はない。 + +未成立・失敗を成功 trace として保存しない。一部 scan source、sparse retrieval、graph expansion の失敗、canonical identity 未解決、memory の原子的書込失敗では検索結果を返せるが、`memory_unavailable:true`、`feedback_available:false` とし、**trace_id を発行しない**。この結果への feedback は不可。拒否 error は段階違反、key conflict、confirmation 等の安全な code を返し、query/handle を error や log へ転記しない。検索の error log は固定文言にする。 + +## 利用段階と retry + +`record_source_use {trace_id,idempotency_key,uses:[{source_id,stage}]}` で段階を記録する。 + +| stage | 意味 | usage activation 加算 | +|---|---|---:| +| selected | 調査対象として選んだ | 0.2 | +| validated | 正確な出典を確認し利用可と判断した | 0.5 | +| used | 回答・判断に実際に使用した | 1.0 | + +段階は一つずつ進める。caller が各判断を済ませた後、同じ資料の3段階を同じ batch に含めてもよい。trace 所属、順序、owner、全 entry を一つの SQLite `transactionSync` で確認し、拒否時は usage・activation・receipt 全てを rollback する。既に完了した段階の再送は no-op。検索自体を used や confirmed と解釈しない。 + +key は principal 全体で mutation tool / trace をまたいで一意。同じ parsed payload の retry は同 receipt、同 key の異なる payload は `idempotency_conflict`。JSON object の field 順序は同一性に影響せず、array 順序は影響する。並行更新は DO と transaction で直列化する。新しい操作には新しい key を使う。 + +## 活性度と学習の反映 + +`MEMORY_POLICY` の半減期は3600秒。読取・更新時に `value * 0.5^(max(0,elapsed_ms)/3600000)` で計算する。clock 逆行は elapsed=0、更新 timestamp は逆行させない。cron は不要。 + +成功 trace ごとに distinct な返却資料へ `retrieved` を0.1加算。`usage` は上記 stage の加算量を使う。**各 channel 上限10、total 上限20**で、有限・非負。応答は両 channel、total、UTC updated_at を返す。取消しは関係の寄与を取り消し、過去の usage 記録や activation は消さない。 + +検索履歴は既定で書き込み、search の `readOnlyHint:false`。keyword の fusion / score / rank / rerank は維持する。`graph_expand:false` が既定。`use_memory:false` は従来の graph 順序・上限を保つ。 + +search mode の `use_memory:true` は現在の graph 候補を最大200件読み、hop 昇順、同 hop 内だけ減衰後の path strength 合計降順で並べ、同点は元順序を保つ。その後 `min(top_k*2,30)` 件へ絞るので、従来の返却枠外の強化済み候補も選べる。初期 strength=0 では従来と同じ順序。両軸を融合せず、relevance score を追加しない。opt-in の graph row に付く `learned_strength` は寄与の合計で、関連度ではない。scan/fetch はこの並べ替え指定を無視する。 + +## confirmed と因果的取消し + +`record_outcome {trace_id,idempotency_key,outcome:"confirmed",source_ids,reason}` は同 trace の used 済み資料を要求する。探索時に保存した実在する1〜2本の directional mention edge だけを強化し、origin→終点の shortcut、lexical/zero-hop edge は作らない。usage だけでは関係を強化しない。 + +独立 trace は保存 edge ごとに0.25を加算し、減衰後 strength の上限は5。同 trace は同じ edge に一度しか寄与できず、資料間の path 重複、再確認、retry、取消し後も二重加算しない。実 delta は残り headroom（0の場合もある）。receipt / ledger は saved path、delta、timestamp、reason、active/reversed を保持する。重複、上限、lexical、取消し済みは非適用理由を返す。 + +`corrected` / `rolled_back` は**同 trace の confirmation_id を一つ明示**し、source_ids を渡さない。取消しはその receipt の寄与だけを inactive にし、現在まで減衰した delta だけを減算する。後続の独立寄与を残し、weight 全体を reset しない。未知・曖昧な confirmation は原子的に拒否。元 receipt と別の取消し receipt を監査に保持する。取消された trace は同 edge の credit を再取得できない。 + +## 移行・deploy・artifact + +この変更では version/tag/release を公開しない。新しい Worker binding / DO class migration は不要。既存 SQLite `IssueStore` に独立 `memory_*` table を追加し、既存 issue table / watermark は保持する。 + +**D1 migration 0008 を Worker deploy、または自動 deploy を起こす branch への merge より先に適用する。** `comment_id` / `review_id` を追加し、0は未解決の旧行を表す。運用段階の具体的コマンドは以下（credential は出力しない）。 + +```powershell +# read-only の事前確認 +npx wrangler d1 migrations list github-rag-fts --remote +npx wrangler d1 execute github-rag-fts --remote --command "PRAGMA table_info(search_docs)" +# 本番変更の許可後、Worker deployment 前に適用 +npx wrangler d1 migrations apply github-rag-fts --remote +# 両列を確認後、review 済み Worker commit を deploy +npx wrangler d1 execute github-rag-fts --remote --command "PRAGMA table_info(search_docs)" +# 0008後の未解決件数（read-only） +npx wrangler d1 execute github-rag-fts --remote --command "SELECT type,COUNT(*) AS unresolved FROM search_docs WHERE (type IN ('issue_comment','pr_review_comment') AND comment_id=0) OR (type='pr_review' AND review_id=0) GROUP BY type" +``` + +deploy 後の認可された `POST /admin/backfill-source-identities?repo=owner/repo&limit=50[&cursor=TYPE:ID]` は、既存 DO の canonical event ID から過去の unchanged 行を補修する。既存管理 credential は非公開 header で渡し、URL / artifact / shell history に token を載せない。next_cursor を渡して done:true まで進める。limit は1..100、1 page は最大3つの有界 DO read と一つの原子的 D1 batch。再開・再実行は安全で、embedding / GitHub 本文再取得 / index reset は不要。 + +通常の unchanged ingest も hash skip の前に ID を補修する。DO に canonical event が無い行は未解決のままなので、別の再取り込み前に欠損を調べる。未解決 source を含む検索は memory を fail-closed にする。backfill は private trace/credit を変更しない。 + +bridge には `server/search-schema.json` と `server/memory-tools.json` を同梱する。`node scripts/generate-tool-contracts.mjs` で Worker Zod contract から生成し、`node scripts/check-schema-drift.mjs` が全 tool の実 protocol と nested 入出力 schema / bounds / defaults / annotations の完全一致を確認する。npm / mcpb の両 artifact にこの JSON が必要。stdio client の tool discovery には更新 bridge の公開が必要。公開 version と publication は後続の運用段階で決める。 + +## 合成 lifecycle と検証 + +`npx vitest run --config vitest.workers.config.ts src/memory-e2e.workers.test.ts src/memory.workers.test.ts` で、synthetic principal と local in-process SQLite/D1 のみを使う。 + +実演は `syntheticneedle` 検索 → path `a->b->c` → `c` の selected/validated/used → 2 edge に各0.25の confirmed → corrected で両寄与 inactive。独立 trace の固定時刻テストでは半減期後に次の確認を加えると path strength=0.75、最初の寄与を取り消すと次の寄与0.50が残る。実 MCP の live 本文回帰では索引 timestamp と keyword 順位を固定し、doc/wiki の inline 本文のみ変更する。同じ本文の retry は同 ID、変更本文は別 ID、stored fetch は index ID を保ち、旧 live 版の usage は新しい版に引き継がれない。正しさの検証であり、検索品質向上の評価ではない。 + +成功検索には DO request を一つと、有界な source hash / activation 処理が加わる。学習 opt-in は最大200 path、各最大2 edge を追加で読む。storage は trace/receipt とともに増え、自動 retention deletion は実装しない。global DO が書込を直列化する。feedback に Python 常駐、モデル配布、追加 embedding、mention topology の書換えは不要。 diff --git a/docs/2-feedback-memory.md b/docs/2-feedback-memory.md new file mode 100644 index 0000000..ddbb475 --- /dev/null +++ b/docs/2-feedback-memory.md @@ -0,0 +1,73 @@ +# Private retrieval feedback memory + +Language: English | [Japanese](2-feedback-memory.ja.md) + +This specification completes Issue #259's four connected functions: retrieval history, explicit source usage, decaying activation, and reversible graph feedback. It stores the query, retrieval settings, returned source provenance and explicit decisions. It does not archive conversations or duplicate document bodies, titles, access credentials, or authentication props. + +## Identity and traces + +Every successful `search` call (search, scan, stored-content fetch, and zero hits) commits a UTC trace before returning `trace_id`, `timestamp`, `feedback_available: true`, `source_id`, `provenance`, and current `activation` on returned rows, including folded `same_entity.others`. The request records filters and controls; settings record mode, effective scan window, fusion/rerank outcomes and both retrieval axes. Source provenance is `{repo,type,identity,version,content_source}`. Identity is issue/PR number, release tag, doc/wiki path, `[commit_sha,file_path]` for diffs, or canonical GitHub comment/review ID. `content_source:"index"` uses captured `updated_at` as `version` (the indexed snapshot timestamp for doc/wiki pages). Live inline doc/wiki bodies carry `content_source:"github_live"` and `content_version:"sha256:"`; that hash is their provenance version and enters the source ID. `index_updated_at` preserves the indexed timestamp as supplemental audit metadata without defining the live version. The same live body stays the same source version when the index timestamp changes; a changed returned body changes its source ID even when the index timestamp is unchanged. Stored-content fetch and graph-axis bodies refer to the indexed copy and keep separate index provenance. Memory stores only the hash/provenance, never the live body. Arbitrary supplemental fields, vector handles and mode do not enter the ID hash. + +`source_id` identifies a particular source version, not the unversioned entity. Same entity and captured version have the same ID after a vector-handle migration; an updated version has a different ID. Activation belongs to that version. Mention strength belongs to canonical `[repo,src_slug,dst_slug,edge_kind]`, independently of vector IDs and document versions. Reindexing mention topology neither resets nor rewrites private learned state. Removed mentions cannot be traversed; a saved historical path remains auditable and reversible. + +The server derives `github:` from verified MCP OAuth props. Caller identity fields are not trusted. Same-user clients share private traces and receipts; other users get `unknown_trace`/`unknown_confirmation` rather than someone else's data. The internal DO interface is not publicly routed. + +`memory_history {limit:10}` returns newest trace summaries and `next_cursor`; pass that cursor for the next page. `memory_history {trace_id,limit:10}` returns sources, current stages/activation, settings and oldest-first receipt audit. Its `next_cursor` pages **receipts within that trace**, not trace summaries. Cursors belong to the authenticated principal and, for details, the trace. Limits are 1..50; source snapshots cap at 300 per trace, saved trace input at 300,000 characters, queries at 4096 characters, metadata strings at 256, usage batches at 100 entries, and reason/idempotency handles at 1000/128 characters. SQL reads use owner/cursor indexes and bounded pages; no full-history fetch is used. + +A failed retrieval is not recorded as successful. If retrieval is partial (a scan surface, sparse retrieval, or graph expansion failed), identity cannot be resolved, or the atomic memory write fails, retrieval can still return with `memory_unavailable: true`, `feedback_available: false` and **no `trace_id`**. It cannot accept feedback. Memory rejection codes describe ownership, stage, key conflict and confirmation errors without echoing the submitted query or handles. Retrieval error logging emits fixed messages. + +## Usage and idempotency + +`record_source_use {trace_id,idempotency_key,uses:[{source_id,stage}]}` records: + +| Stage | Meaning | Activation addition | +|---|---|---:| +| selected | Chosen for investigation | 0.2 | +| validated | Exact source checked and judged usable | 0.5 | +| used | Actually used in an answer or decision | 1.0 | + +Stages advance one step at a time. A batch can contain all three steps for the same source after the caller has performed them. Trace membership, stage order, owner and every batch entry are checked in one SQLite `transactionSync`; any rejection rolls back usage, activation and receipt. Resending an already completed stage is a no-op. Retrieval alone is not usage or confirmation. + +Idempotency keys are principal-wide across mutation tools and traces. A retry with the same parsed payload returns the original receipt. Another payload with that key returns `idempotency_conflict`. JSON object field order does not affect equivalence; array order does. Concurrent requests serialize through the DO and transaction. A new operation needs a new key. + +## Activation and learned strength + +`MEMORY_POLICY` fixes half-life at 3600 seconds. On reads/updates, value decays as `value * 0.5^(max(0,elapsed_ms)/3600000)`. Backwards time contributes zero elapsed, and updates keep timestamps monotonic. No cron is needed. `retrieved` adds 0.1 per distinct returned source per successful trace; `usage` adds the explicit stage amounts above. Each channel caps at 10, total at 20; values are finite and nonnegative. Readback includes both channels, their total, and UTC `updated_at`. Reversal removes relation credit; it does not erase a historical usage or its activation. + +Search writes history by default and has `readOnlyHint:false`. Keyword fusion, scores, ranks and rerank behavior are unchanged. `graph_expand:false` remains the graph default. `use_memory:false` preserves the existing graph output order/limit. `use_memory:true` in search mode orders a bounded pool of up to 200 current graph candidates by hop ascending, then summed decayed path strength descending, preserving original order on ties; it then returns at most `min(top_k*2,30)`. This can admit a learned candidate outside the previous output cap. Initial zero strength yields the existing order. Learning never mixes the axes or adds a relevance score; only opt-in graph rows carry `learned_strength` (sum of relation credit, not relevance). Scan/fetch ignore this ordering switch. + +## Confirmations and causal reversal + +`record_outcome {trace_id,idempotency_key,outcome:"confirmed",source_ids,reason}` requires used sources from that trace. It credits only the actual one/two directional mention edges saved by the graph traversal. No origin-to-terminal shortcut, lexical edge or zero-hop edge is created. Usage alone never strengthens a relation. + +Each independent trace can add 0.25 per saved edge, with a decayed edge-strength cap of 5. A trace contributes to an edge only once, even when multiple returned sources share it, confirmation is repeated, or its old contribution was reversed. The applied delta is the remaining cap headroom (possibly zero). Receipt and ledger save path, delta, timestamp, reason and active/reversed status. Duplicate, cap, lexical, and already-reversed operations expose non-application reasons. + +`corrected`/`rolled_back` requires **one explicit `confirmation_id` from the same trace** and omits `source_ids`. Reversal deactivates precisely that receipt's contributions and subtracts their present decayed deltas. Later independent contributions remain; no scalar reset or inverse guess is used. Unknown/ambiguous confirmations reject atomically. Audit preserves the original receipt and a separate reversal receipt. Reversed traces cannot reacquire credit for the same edge. + +## Deployment and schema preparation + +No version, tag or public release is created by this change. No new Worker binding or DO class migration is needed: independent `memory_*` tables are created additively inside existing SQLite `IssueStore`. Existing issue tables and watermarks are retained. + +Apply D1 migration **0008 before deploying or merging into a branch that automatically deploys**. It adds `comment_id`/`review_id`, with zero as the unresolved legacy sentinel. Deployment preparation (operator action; do not publish credentials): + +```powershell +# Read-only preflight +npx wrangler d1 migrations list github-rag-fts --remote +npx wrangler d1 execute github-rag-fts --remote --command "PRAGMA table_info(search_docs)" +# After the operator authorizes production mutation, before Worker deployment +npx wrangler d1 migrations apply github-rag-fts --remote +# Verify both new columns; then deploy the reviewed Worker commit +npx wrangler d1 execute github-rag-fts --remote --command "PRAGMA table_info(search_docs)" +# Read-only unresolved-row count (after 0008) +npx wrangler d1 execute github-rag-fts --remote --command "SELECT type,COUNT(*) AS unresolved FROM search_docs WHERE (type IN ('issue_comment','pr_review_comment') AND comment_id=0) OR (type='pr_review' AND review_id=0) GROUP BY type" +``` + +After deployment, authorized `POST /admin/backfill-source-identities?repo=owner/repo&limit=50[&cursor=TYPE:ID]` repairs historical unchanged rows from existing canonical DO events. Supply the existing administrative credential through a private header; no token belongs in a URL, artifact or shell history. Follow `next_cursor` until `done:true`. Limit 1..100, at most three bounded DO reads and one atomic D1 batch per page. Restarting is safe; no embeddings, GitHub body refetch or index reset is needed. Normal unchanged comment/review ingest also repairs IDs before its hash-skip return. Rows whose canonical event is absent from the DO remain unresolved; inspect that gap before any separate reingest. A trace containing unresolved source identity fails memory closed. Backfill does not rewrite private traces/credits. + +The bridge includes `server/search-schema.json` and `server/memory-tools.json`. `node scripts/generate-tool-contracts.mjs` regenerates them from Worker Zod schemas; `node scripts/check-schema-drift.mjs` compares all shipped tools' exact nested input/output contracts, bounds, defaults and annotations against the actual protocol. Package both JSON files in npm/mcpb artifacts. Worker deployment enables direct-client tools; the published bridge must include the new artifacts for stdio clients to discover them. Public version selection and publication are subsequent operations. + +## Verification and synthetic example + +Run `npx vitest run --config vitest.workers.config.ts src/memory-e2e.workers.test.ts src/memory.workers.test.ts`. These fixtures use synthetic principals and local in-process SQLite/D1. The observed lifecycle is search of `syntheticneedle` -> graph path `a->b->c` -> selected/validated/used for `c` -> confirmation with two 0.25 deltas -> corrected with both credits inactive. The independent-trace test confirms again one half-life later: path strength 0.75 -> reversing the first trace leaves 0.50 from the second. The real MCP live-body regression changes only doc/wiki inline text while holding the index timestamp and keyword ranks fixed: retry keeps the same ID, changed live text gets another ID, stored fetch retains the index ID, and usage of the old live version does not mark the new version used. This is correctness evidence, not an evaluation that retrieval quality improved. + +Cost: each successful retrieval adds one DO request and bounded source hash/activation work. Opt-in learning additionally reads up to 200 saved paths (up to two edges each). Storage grows with private traces and receipts; no automatic retention deletion is included. This global DO serializes writes. No Python process, model distribution, embedding step, or extra graph topology write is added by feedback. diff --git a/mcp-server/README.md b/mcp-server/README.md index f79a91b..bd2956a 100644 --- a/mcp-server/README.md +++ b/mcp-server/README.md @@ -103,7 +103,7 @@ Delete these files to force a fresh authorization flow. ## Tools exposed -A single consolidated tool, `search`, covers every retrieval mode. Earlier builds split these across `get_issue_context`, `get_doc_content`, and `list_recent_activity`; those tools have been removed and their use cases now fold into the parameters of `search`. +`search` covers every retrieval mode, alongside private `memory_history`, `record_source_use`, and `record_outcome` tools. Earlier builds split these across `get_issue_context`, `get_doc_content`, and `list_recent_activity`; those tools have been removed and their use cases now fold into the parameters of `search`. | Tool | Description | |---|---| @@ -111,7 +111,7 @@ A single consolidated tool, `search`, covers every retrieval mode. Earlier build The `type` filter accepts: `issue`, `pull_request`, `release`, `doc`, `wiki_doc`, `diff`, `issue_comment`, `pr_review`, `pr_review_comment`, or `all` (default). -`search` is read-only. For the full parameter reference and examples, see the [main repository README](https://github.com/Liplus-Project/github-rag-mcp#search). +`search` writes private retrieval history and has `readOnlyHint:false`. The three feedback tools provide bounded history, explicit selected/validated/used records, and confirmed path reinforcement with receipt-based correction/rollback. `use_memory` defaults false and only changes same-hop graph ordering. Include `server/search-schema.json` and `server/memory-tools.json` in shipped artifacts. See the [memory specification](https://github.com/Liplus-Project/github-rag-mcp/blob/main/docs/2-feedback-memory.md) for bounds, costs, migration 0008-before-deployment and the synthetic lifecycle. For the full parameter reference and examples, see the [main repository README](https://github.com/Liplus-Project/github-rag-mcp#search). ## Authentication flow @@ -161,3 +161,5 @@ The browser callback never leaves your machine; the authorization code is delive ## License Apache-2.0. See the [LICENSE](https://github.com/Liplus-Project/github-rag-mcp/blob/main/LICENSE) and [NOTICE](https://github.com/Liplus-Project/github-rag-mcp/blob/main/NOTICE) files in the main repository. + +Live inline doc/wiki bodies carry `github_live` provenance and a SHA-256 content version; stored fetch and graph bodies retain indexed-snapshot provenance. The source IDs distinguish those versions without archiving the live body. diff --git a/mcp-server/manifest.json b/mcp-server/manifest.json index 89f2b4f..66ded79 100644 --- a/mcp-server/manifest.json +++ b/mcp-server/manifest.json @@ -4,7 +4,7 @@ "display_name": "GitHub RAG MCP", "version": "0.1.0", "description": "Hybrid semantic search across GitHub issues, PRs, releases, docs, GitHub Wiki pages, commit diffs, and review threads via Cloudflare Worker + Vectorize.", - "long_description": "GitHub RAG MCP is a memory-DB surface over GitHub project state: issues, PRs, releases, repository docs, GitHub Wiki pages, commit diffs, issue/PR top-level comments, PR reviews, and PR inline review comments are indexed into Cloudflare Vectorize (BGE-M3 dense) + D1 FTS5 (BM25 sparse) and fused via RRF + cross-encoder rerank. A single consolidated tool (search) handles hybrid semantic search, time-ordered activity scan, and inline doc / wiki content fetch, selected via its query / sort / include_content axes. Counterpart to github-webhook-mcp (push-based notifications) — together they give AI a complete view of GitHub project state.", + "long_description": "GitHub RAG MCP is a memory-DB surface over GitHub project state: issues, PRs, releases, repository docs, GitHub Wiki pages, commit diffs, issue/PR top-level comments, PR reviews, and PR inline review comments are indexed into Cloudflare Vectorize (BGE-M3 dense) + D1 FTS5 (BM25 sparse) and fused via RRF + cross-encoder rerank. A single consolidated tool (search) handles hybrid semantic search, time-ordered activity scan, and inline doc / wiki content fetch, selected via its query / sort / include_content axes. Counterpart to github-webhook-mcp (push-based notifications) — together they give AI a complete view of GitHub project state.\nPrivate retrieval traces, selected/validated/used source records, decaying activation, and confirmed-only graph feedback with receipt-based correction/rollback. Learning is opt-in via use_memory and preserves keyword ranks.", "author": { "name": "Liplus Project", "url": "https://github.com/Liplus-Project" @@ -43,6 +43,18 @@ { "name": "search", "description": "Unified search across GitHub issues, PRs, releases, docs, GitHub Wiki pages, commit diffs, issue/PR top-level comments, PR reviews, and PR inline review comments. Four modes derived from the parameter set: (1) hybrid semantic search (dense BGE-M3 + sparse BM25 fused via RRF + cross-encoder rerank), (2) time-ordered activity scan (empty query + sort), (3) doc/wiki content fetch (include_content=true inlines raw markup for top doc and wiki_doc results), (4) stored-content fetch (vector_ids returns the body text the index holds for the named rows, any type, no GitHub API call, truncated at the 8000-character ingest ceiling). Structured filters: repo, state, labels, milestone, assignee, type (issue / pull_request / release / doc / wiki_doc / diff / issue_comment / pr_review / pr_review_comment)." + }, + { + "name": "memory_history", + "description": "Read private bounded trace history and audit." + }, + { + "name": "record_source_use", + "description": "Record selected, validated, then used source stages atomically." + }, + { + "name": "record_outcome", + "description": "Confirm saved graph paths or reverse a named confirmation receipt." } ], "compatibility": { diff --git a/mcp-server/server/index.js b/mcp-server/server/index.js index 8e0f151..0884b88 100644 --- a/mcp-server/server/index.js +++ b/mcp-server/server/index.js @@ -32,7 +32,7 @@ import { import { createRequire } from "node:module"; import { createOAuthProvider, OAuthPendingError } from "./oauth.js"; import { createRemoteClient } from "./remote-client.js"; -import { TOOLS } from "./tools.js"; +import { TOOLS, MEMORY_CONTRACT } from "./tools.js"; const require = createRequire(import.meta.url); const { version: PACKAGE_VERSION } = require("../package.json"); @@ -67,7 +67,7 @@ async function callRemoteTool(name, args) { const server = new Server( { name: "github-rag-mcp", version: PACKAGE_VERSION }, - { capabilities: { tools: {} } }, + { capabilities: { tools: {} }, instructions: MEMORY_CONTRACT.instructions }, ); // ── Tool Definitions ───────────────────────────────────────────────────────── @@ -93,7 +93,7 @@ server.setRequestHandler(CallToolRequestSchema, async (req) => { }; } return { - content: [{ type: "text", text: `Failed to reach worker: ${err}` }], + content: [{ type: "text", text: "Failed to reach worker" }], isError: true, }; } diff --git a/mcp-server/server/memory-tools.json b/mcp-server/server/memory-tools.json new file mode 100644 index 0000000..6e383c7 --- /dev/null +++ b/mcp-server/server/memory-tools.json @@ -0,0 +1,1866 @@ +{ + "instructions": "Successful search/scan/fetch writes a private trace. Retrieval is not usage or confirmation. Call record_source_use with selected (chosen for investigation), then validated (exact source checked and judged usable), then used (actually used in an answer or decision). record_outcome confirmed reinforces only saved graph paths of used sources. corrected/rolled_back names the confirmation receipt to reverse. Use a new idempotency_key for each operation; retry the same payload with the same key. memory_history lists traces or reads one trace. use_memory only orders graph candidates within the same hop; keyword scores/ranks never change. memory_unavailable means no usable trace was saved. Live inline doc/wiki bodies use returned-content SHA-256 versions and github_live provenance; stored-content fetch refers to the indexed snapshot.", + "tools": [ + { + "name": "memory_history", + "description": "Read your private, bounded trace history. trace_id returns source usage, current activation, saved paths, and outcome/reversal audit. Otherwise limit/cursor pages trace summaries.", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "trace_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "limit": { + "default": 10, + "type": "integer", + "minimum": 1, + "maximum": 50 + }, + "cursor": { + "type": "string", + "minLength": 1, + "maxLength": 128 + } + }, + "additionalProperties": false + }, + "outputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "data": { + "anyOf": [ + { + "type": "object", + "properties": { + "traces": { + "maxItems": 50, + "type": "array", + "items": { + "type": "object", + "properties": { + "trace_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "timestamp": { + "type": "string" + }, + "request": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + }, + "settings": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + } + }, + "required": [ + "trace_id", + "timestamp", + "request", + "settings" + ], + "additionalProperties": false + } + }, + "next_cursor": { + "anyOf": [ + { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + { + "type": "null" + } + ] + } + }, + "required": [ + "traces", + "next_cursor" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "trace_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "timestamp": { + "type": "string" + }, + "request": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + }, + "settings": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + }, + "policy": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + }, + "sources": { + "maxItems": 300, + "type": "array", + "items": { + "type": "object", + "properties": { + "source_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "provenance": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + }, + "axes": { + "type": "array", + "items": { + "type": "string" + } + }, + "path": { + "maxItems": 2, + "type": "array", + "items": { + "type": "object", + "properties": { + "repo": { + "type": "string" + }, + "src": { + "type": "string" + }, + "dst": { + "type": "string" + }, + "kind": { + "type": "string" + } + }, + "required": [ + "repo", + "src", + "dst", + "kind" + ], + "additionalProperties": false + } + }, + "stage": { + "type": "string", + "enum": [ + "retrieved", + "selected", + "validated", + "used" + ] + }, + "activation": { + "type": "object", + "properties": { + "retrieved": { + "type": "number", + "minimum": 0 + }, + "usage": { + "type": "number", + "minimum": 0 + }, + "total": { + "type": "number", + "minimum": 0 + }, + "updated_at": { + "type": "string" + } + }, + "required": [ + "retrieved", + "usage", + "total", + "updated_at" + ], + "additionalProperties": false + } + }, + "required": [ + "source_id", + "provenance", + "axes", + "path", + "stage", + "activation" + ], + "additionalProperties": false + } + }, + "audit": { + "maxItems": 50, + "type": "array", + "items": { + "type": "object", + "properties": { + "receipt_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "trace_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "timestamp": { + "type": "string" + }, + "outcome": { + "type": "string", + "enum": [ + "confirmed", + "corrected", + "rolled_back" + ] + }, + "reason": { + "type": "string" + }, + "confirmation_id": { + "anyOf": [ + { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + { + "type": "null" + } + ] + }, + "uses": { + "type": "array", + "items": { + "type": "object", + "properties": { + "source_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "stage": { + "type": "string", + "enum": [ + "selected", + "validated", + "used" + ] + } + }, + "required": [ + "source_id", + "stage" + ], + "additionalProperties": false + } + }, + "saved_sources": { + "type": "array", + "items": { + "type": "object", + "properties": { + "source_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "provenance": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + }, + "axes": { + "type": "array", + "items": { + "type": "string" + } + }, + "path": { + "maxItems": 2, + "type": "array", + "items": { + "type": "object", + "properties": { + "repo": { + "type": "string" + }, + "src": { + "type": "string" + }, + "dst": { + "type": "string" + }, + "kind": { + "type": "string" + } + }, + "required": [ + "repo", + "src", + "dst", + "kind" + ], + "additionalProperties": false + } + } + }, + "required": [ + "source_id", + "provenance", + "axes", + "path" + ], + "additionalProperties": false + } + }, + "changes": { + "type": "array", + "items": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + } + }, + "non_applied_reason": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "active_credits": { + "type": "array", + "items": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + } + } + }, + "required": [ + "receipt_id", + "trace_id", + "timestamp", + "saved_sources", + "changes", + "non_applied_reason" + ], + "additionalProperties": false + } + }, + "next_cursor": { + "anyOf": [ + { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + { + "type": "null" + } + ] + } + }, + "required": [ + "trace_id", + "timestamp", + "request", + "settings", + "policy", + "sources", + "audit", + "next_cursor" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "receipt_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "trace_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "timestamp": { + "type": "string" + }, + "outcome": { + "type": "string", + "enum": [ + "confirmed", + "corrected", + "rolled_back" + ] + }, + "reason": { + "type": "string" + }, + "confirmation_id": { + "anyOf": [ + { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + { + "type": "null" + } + ] + }, + "uses": { + "type": "array", + "items": { + "type": "object", + "properties": { + "source_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "stage": { + "type": "string", + "enum": [ + "selected", + "validated", + "used" + ] + } + }, + "required": [ + "source_id", + "stage" + ], + "additionalProperties": false + } + }, + "saved_sources": { + "type": "array", + "items": { + "type": "object", + "properties": { + "source_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "provenance": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + }, + "axes": { + "type": "array", + "items": { + "type": "string" + } + }, + "path": { + "maxItems": 2, + "type": "array", + "items": { + "type": "object", + "properties": { + "repo": { + "type": "string" + }, + "src": { + "type": "string" + }, + "dst": { + "type": "string" + }, + "kind": { + "type": "string" + } + }, + "required": [ + "repo", + "src", + "dst", + "kind" + ], + "additionalProperties": false + } + } + }, + "required": [ + "source_id", + "provenance", + "axes", + "path" + ], + "additionalProperties": false + } + }, + "changes": { + "type": "array", + "items": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + } + }, + "non_applied_reason": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "active_credits": { + "type": "array", + "items": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + } + } + }, + "required": [ + "receipt_id", + "trace_id", + "timestamp", + "saved_sources", + "changes", + "non_applied_reason" + ], + "additionalProperties": false + } + ] + } + }, + "required": [ + "data" + ], + "additionalProperties": false + }, + "annotations": { + "readOnlyHint": true, + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false + } + }, + { + "name": "record_source_use", + "description": "Atomically record selected -> validated -> used for sources in your trace. selected = chosen for investigation; validated = exact source checked and usable; used = actually used in an answer/decision. Search alone is never used. Same key/payload returns the saved receipt; conflicting payload is rejected. Repeating a completed stage adds no activation.", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "trace_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "idempotency_key": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "uses": { + "minItems": 1, + "maxItems": 100, + "type": "array", + "items": { + "type": "object", + "properties": { + "source_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "stage": { + "type": "string", + "enum": [ + "selected", + "validated", + "used" + ] + } + }, + "required": [ + "source_id", + "stage" + ], + "additionalProperties": false + } + } + }, + "required": [ + "trace_id", + "idempotency_key", + "uses" + ], + "additionalProperties": false + }, + "outputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "data": { + "anyOf": [ + { + "type": "object", + "properties": { + "traces": { + "maxItems": 50, + "type": "array", + "items": { + "type": "object", + "properties": { + "trace_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "timestamp": { + "type": "string" + }, + "request": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + }, + "settings": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + } + }, + "required": [ + "trace_id", + "timestamp", + "request", + "settings" + ], + "additionalProperties": false + } + }, + "next_cursor": { + "anyOf": [ + { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + { + "type": "null" + } + ] + } + }, + "required": [ + "traces", + "next_cursor" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "trace_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "timestamp": { + "type": "string" + }, + "request": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + }, + "settings": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + }, + "policy": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + }, + "sources": { + "maxItems": 300, + "type": "array", + "items": { + "type": "object", + "properties": { + "source_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "provenance": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + }, + "axes": { + "type": "array", + "items": { + "type": "string" + } + }, + "path": { + "maxItems": 2, + "type": "array", + "items": { + "type": "object", + "properties": { + "repo": { + "type": "string" + }, + "src": { + "type": "string" + }, + "dst": { + "type": "string" + }, + "kind": { + "type": "string" + } + }, + "required": [ + "repo", + "src", + "dst", + "kind" + ], + "additionalProperties": false + } + }, + "stage": { + "type": "string", + "enum": [ + "retrieved", + "selected", + "validated", + "used" + ] + }, + "activation": { + "type": "object", + "properties": { + "retrieved": { + "type": "number", + "minimum": 0 + }, + "usage": { + "type": "number", + "minimum": 0 + }, + "total": { + "type": "number", + "minimum": 0 + }, + "updated_at": { + "type": "string" + } + }, + "required": [ + "retrieved", + "usage", + "total", + "updated_at" + ], + "additionalProperties": false + } + }, + "required": [ + "source_id", + "provenance", + "axes", + "path", + "stage", + "activation" + ], + "additionalProperties": false + } + }, + "audit": { + "maxItems": 50, + "type": "array", + "items": { + "type": "object", + "properties": { + "receipt_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "trace_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "timestamp": { + "type": "string" + }, + "outcome": { + "type": "string", + "enum": [ + "confirmed", + "corrected", + "rolled_back" + ] + }, + "reason": { + "type": "string" + }, + "confirmation_id": { + "anyOf": [ + { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + { + "type": "null" + } + ] + }, + "uses": { + "type": "array", + "items": { + "type": "object", + "properties": { + "source_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "stage": { + "type": "string", + "enum": [ + "selected", + "validated", + "used" + ] + } + }, + "required": [ + "source_id", + "stage" + ], + "additionalProperties": false + } + }, + "saved_sources": { + "type": "array", + "items": { + "type": "object", + "properties": { + "source_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "provenance": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + }, + "axes": { + "type": "array", + "items": { + "type": "string" + } + }, + "path": { + "maxItems": 2, + "type": "array", + "items": { + "type": "object", + "properties": { + "repo": { + "type": "string" + }, + "src": { + "type": "string" + }, + "dst": { + "type": "string" + }, + "kind": { + "type": "string" + } + }, + "required": [ + "repo", + "src", + "dst", + "kind" + ], + "additionalProperties": false + } + } + }, + "required": [ + "source_id", + "provenance", + "axes", + "path" + ], + "additionalProperties": false + } + }, + "changes": { + "type": "array", + "items": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + } + }, + "non_applied_reason": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "active_credits": { + "type": "array", + "items": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + } + } + }, + "required": [ + "receipt_id", + "trace_id", + "timestamp", + "saved_sources", + "changes", + "non_applied_reason" + ], + "additionalProperties": false + } + }, + "next_cursor": { + "anyOf": [ + { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + { + "type": "null" + } + ] + } + }, + "required": [ + "trace_id", + "timestamp", + "request", + "settings", + "policy", + "sources", + "audit", + "next_cursor" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "receipt_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "trace_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "timestamp": { + "type": "string" + }, + "outcome": { + "type": "string", + "enum": [ + "confirmed", + "corrected", + "rolled_back" + ] + }, + "reason": { + "type": "string" + }, + "confirmation_id": { + "anyOf": [ + { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + { + "type": "null" + } + ] + }, + "uses": { + "type": "array", + "items": { + "type": "object", + "properties": { + "source_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "stage": { + "type": "string", + "enum": [ + "selected", + "validated", + "used" + ] + } + }, + "required": [ + "source_id", + "stage" + ], + "additionalProperties": false + } + }, + "saved_sources": { + "type": "array", + "items": { + "type": "object", + "properties": { + "source_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "provenance": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + }, + "axes": { + "type": "array", + "items": { + "type": "string" + } + }, + "path": { + "maxItems": 2, + "type": "array", + "items": { + "type": "object", + "properties": { + "repo": { + "type": "string" + }, + "src": { + "type": "string" + }, + "dst": { + "type": "string" + }, + "kind": { + "type": "string" + } + }, + "required": [ + "repo", + "src", + "dst", + "kind" + ], + "additionalProperties": false + } + } + }, + "required": [ + "source_id", + "provenance", + "axes", + "path" + ], + "additionalProperties": false + } + }, + "changes": { + "type": "array", + "items": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + } + }, + "non_applied_reason": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "active_credits": { + "type": "array", + "items": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + } + } + }, + "required": [ + "receipt_id", + "trace_id", + "timestamp", + "saved_sources", + "changes", + "non_applied_reason" + ], + "additionalProperties": false + } + ] + } + }, + "required": [ + "data" + ], + "additionalProperties": false + }, + "annotations": { + "readOnlyHint": false, + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false + } + }, + { + "name": "record_outcome", + "description": "confirmed requires source_ids already used in this trace and strengthens only saved actual graph paths. One trace contributes to each edge once. corrected/rolled_back requires confirmation_id and reverses only that receipt contribution, preserving other traces. reason is saved for audit. Lexical sources have no graph credit. Idempotent by key/payload.", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "trace_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "idempotency_key": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "outcome": { + "type": "string", + "enum": [ + "confirmed", + "corrected", + "rolled_back" + ] + }, + "source_ids": { + "minItems": 1, + "maxItems": 100, + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "maxLength": 128 + } + }, + "confirmation_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "reason": { + "type": "string", + "minLength": 1, + "maxLength": 1000 + } + }, + "required": [ + "trace_id", + "idempotency_key", + "outcome", + "reason" + ], + "additionalProperties": false + }, + "outputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "data": { + "anyOf": [ + { + "type": "object", + "properties": { + "traces": { + "maxItems": 50, + "type": "array", + "items": { + "type": "object", + "properties": { + "trace_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "timestamp": { + "type": "string" + }, + "request": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + }, + "settings": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + } + }, + "required": [ + "trace_id", + "timestamp", + "request", + "settings" + ], + "additionalProperties": false + } + }, + "next_cursor": { + "anyOf": [ + { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + { + "type": "null" + } + ] + } + }, + "required": [ + "traces", + "next_cursor" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "trace_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "timestamp": { + "type": "string" + }, + "request": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + }, + "settings": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + }, + "policy": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + }, + "sources": { + "maxItems": 300, + "type": "array", + "items": { + "type": "object", + "properties": { + "source_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "provenance": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + }, + "axes": { + "type": "array", + "items": { + "type": "string" + } + }, + "path": { + "maxItems": 2, + "type": "array", + "items": { + "type": "object", + "properties": { + "repo": { + "type": "string" + }, + "src": { + "type": "string" + }, + "dst": { + "type": "string" + }, + "kind": { + "type": "string" + } + }, + "required": [ + "repo", + "src", + "dst", + "kind" + ], + "additionalProperties": false + } + }, + "stage": { + "type": "string", + "enum": [ + "retrieved", + "selected", + "validated", + "used" + ] + }, + "activation": { + "type": "object", + "properties": { + "retrieved": { + "type": "number", + "minimum": 0 + }, + "usage": { + "type": "number", + "minimum": 0 + }, + "total": { + "type": "number", + "minimum": 0 + }, + "updated_at": { + "type": "string" + } + }, + "required": [ + "retrieved", + "usage", + "total", + "updated_at" + ], + "additionalProperties": false + } + }, + "required": [ + "source_id", + "provenance", + "axes", + "path", + "stage", + "activation" + ], + "additionalProperties": false + } + }, + "audit": { + "maxItems": 50, + "type": "array", + "items": { + "type": "object", + "properties": { + "receipt_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "trace_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "timestamp": { + "type": "string" + }, + "outcome": { + "type": "string", + "enum": [ + "confirmed", + "corrected", + "rolled_back" + ] + }, + "reason": { + "type": "string" + }, + "confirmation_id": { + "anyOf": [ + { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + { + "type": "null" + } + ] + }, + "uses": { + "type": "array", + "items": { + "type": "object", + "properties": { + "source_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "stage": { + "type": "string", + "enum": [ + "selected", + "validated", + "used" + ] + } + }, + "required": [ + "source_id", + "stage" + ], + "additionalProperties": false + } + }, + "saved_sources": { + "type": "array", + "items": { + "type": "object", + "properties": { + "source_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "provenance": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + }, + "axes": { + "type": "array", + "items": { + "type": "string" + } + }, + "path": { + "maxItems": 2, + "type": "array", + "items": { + "type": "object", + "properties": { + "repo": { + "type": "string" + }, + "src": { + "type": "string" + }, + "dst": { + "type": "string" + }, + "kind": { + "type": "string" + } + }, + "required": [ + "repo", + "src", + "dst", + "kind" + ], + "additionalProperties": false + } + } + }, + "required": [ + "source_id", + "provenance", + "axes", + "path" + ], + "additionalProperties": false + } + }, + "changes": { + "type": "array", + "items": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + } + }, + "non_applied_reason": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "active_credits": { + "type": "array", + "items": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + } + } + }, + "required": [ + "receipt_id", + "trace_id", + "timestamp", + "saved_sources", + "changes", + "non_applied_reason" + ], + "additionalProperties": false + } + }, + "next_cursor": { + "anyOf": [ + { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + { + "type": "null" + } + ] + } + }, + "required": [ + "trace_id", + "timestamp", + "request", + "settings", + "policy", + "sources", + "audit", + "next_cursor" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "receipt_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "trace_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "timestamp": { + "type": "string" + }, + "outcome": { + "type": "string", + "enum": [ + "confirmed", + "corrected", + "rolled_back" + ] + }, + "reason": { + "type": "string" + }, + "confirmation_id": { + "anyOf": [ + { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + { + "type": "null" + } + ] + }, + "uses": { + "type": "array", + "items": { + "type": "object", + "properties": { + "source_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "stage": { + "type": "string", + "enum": [ + "selected", + "validated", + "used" + ] + } + }, + "required": [ + "source_id", + "stage" + ], + "additionalProperties": false + } + }, + "saved_sources": { + "type": "array", + "items": { + "type": "object", + "properties": { + "source_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "provenance": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + }, + "axes": { + "type": "array", + "items": { + "type": "string" + } + }, + "path": { + "maxItems": 2, + "type": "array", + "items": { + "type": "object", + "properties": { + "repo": { + "type": "string" + }, + "src": { + "type": "string" + }, + "dst": { + "type": "string" + }, + "kind": { + "type": "string" + } + }, + "required": [ + "repo", + "src", + "dst", + "kind" + ], + "additionalProperties": false + } + } + }, + "required": [ + "source_id", + "provenance", + "axes", + "path" + ], + "additionalProperties": false + } + }, + "changes": { + "type": "array", + "items": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + } + }, + "non_applied_reason": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "active_credits": { + "type": "array", + "items": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + } + } + }, + "required": [ + "receipt_id", + "trace_id", + "timestamp", + "saved_sources", + "changes", + "non_applied_reason" + ], + "additionalProperties": false + } + ] + } + }, + "required": [ + "data" + ], + "additionalProperties": false + }, + "annotations": { + "readOnlyHint": false, + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false + } + } + ] +} diff --git a/mcp-server/server/search-schema.json b/mcp-server/server/search-schema.json new file mode 100644 index 0000000..44c8f34 --- /dev/null +++ b/mcp-server/server/search-schema.json @@ -0,0 +1,208 @@ +{ + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "query": { + "description": "Natural language search query. When omitted or empty, the tool switches to metadata-only scan mode and results are ordered by the timestamp implied by sort (default sort=\"updated_desc\" for empty query).", + "type": "string", + "maxLength": 4096 + }, + "repo": { + "description": "Filter by repository — full slug (owner/repo), exact match. A bare repository name (\"my-repo\") matches nothing and yields an empty result set; search mode flags that case as \"repo\" in the response's filters_unmatched.", + "type": "string", + "maxLength": 256 + }, + "path_prefix": { + "description": "Filter repository docs by a repository-relative directory prefix before ranking. Requires type=\"doc\", a trailing /, no leading /, backslash, NUL, empty, . or .. path segments, and at most 64 UTF-8 bytes. Search mode reports an unmatched value as \"path_prefix\" in filters_unmatched.", + "type": "string", + "maxLength": 256 + }, + "state": { + "default": "all", + "description": "Filter by state", + "type": "string", + "enum": [ + "open", + "closed", + "all" + ] + }, + "labels": { + "description": "Filter by label names (AND logic)", + "maxItems": 50, + "type": "array", + "items": { + "type": "string", + "maxLength": 256 + } + }, + "milestone": { + "description": "Filter by milestone title", + "type": "string", + "maxLength": 256 + }, + "assignee": { + "description": "Filter by assignee login", + "type": "string", + "maxLength": 256 + }, + "type": { + "default": "all", + "description": "Filter by type (default: all). \"doc\" = repository docs (files in /docs/ etc.). \"wiki_doc\" = GitHub Wiki pages (separate from repo docs; both surfaces co-exist). \"diff\" = per-file commit diffs. \"issue_comment\" = top-level comments on issues and PRs. \"pr_review\" = PR review bodies (approve / request_changes / comment). \"pr_review_comment\" = inline per-line review comments on PR diffs.", + "type": "string", + "enum": [ + "issue", + "pull_request", + "release", + "doc", + "wiki_doc", + "diff", + "issue_comment", + "pr_review", + "pr_review_comment", + "all" + ] + }, + "top_k": { + "default": 10, + "description": "Max results (default: 10, max: 50)", + "type": "integer", + "minimum": 1, + "maximum": 50 + }, + "fusion": { + "default": "rrf", + "description": "Fusion strategy. Default: rrf (Reciprocal Rank Fusion over dense + sparse). dense_only = Vectorize only. sparse_only = D1 FTS5 BM25 only. Use rrf unless debugging a specific ranker. Ignored in metadata-only scan mode (empty query).", + "type": "string", + "enum": [ + "rrf", + "dense_only", + "sparse_only" + ] + }, + "rerank": { + "default": true, + "description": "Cross-encoder reranking with @cf/baai/bge-reranker-base. Default: true. When enabled, the fused (or single-ranker) candidates are overfetched (top_k × 5, max 50), post-filtered, then re-scored by the cross-encoder before being trimmed to top_k. Set false to disable (faster, no Workers AI rerank cost; recommended for debugging or when query is a short identifier where lexical match is already decisive). Ignored in metadata-only scan mode (empty query).", + "type": "boolean" + }, + "sort": { + "description": "Result ordering. Default: \"relevance\" when query is non-empty, \"updated_desc\" when query is empty. \"updated_desc\" / \"created_desc\" force time-ordered output and override ranker scores.", + "type": "string", + "enum": [ + "relevance", + "updated_desc", + "created_desc" + ] + }, + "since": { + "description": "ISO 8601 timestamp (inclusive) — keep only results whose updated_at >= since. Pair with sort=\"updated_desc\" + empty query for an activity scan. Default in scan mode: 7 days back from until (or from now when until is omitted).", + "type": "string", + "maxLength": 256 + }, + "until": { + "description": "ISO 8601 timestamp (exclusive) — keep only results whose updated_at < until. In scan mode the [since, until) window is applied inside the index, so any window holding rows returns rows however far back it sits; the response carries truncated: true when the window holds more than one page.", + "type": "string", + "maxLength": 256 + }, + "include_content": { + "default": false, + "description": "When true, inline live doc bodies from the GitHub contents API and wiki_doc bodies from raw wiki URLs. Successful live bodies carry content_source=github_live and content_version=SHA-256 of the returned text; their source_id identifies this live version, separately from the indexed snapshot. Capped at 5 rows of each type. Non-doc rows are unaffected. Graph-axis bodies remain the indexed copy, marked content_source=index.", + "type": "boolean" + }, + "vector_ids": { + "description": "Stored-content fetch. Pass the vector_id values carried by earlier search-mode results (scan-mode rows come from the structured store and carry none) to read back the body text the index holds for those exact rows, for every type — issue, pull_request, issue_comment, pr_review, pr_review_comment, release, diff, doc, wiki_doc. Served from D1: no GitHub API call is made. Takes precedence over the other modes — query, sort, and every metadata filter are ignored when this is present, because the rows are named rather than selected. The text is the INDEXED copy of the body (the embedding input), truncated at 8000 characters — not the live source. Each row carries content_truncated so a prefix is never mistaken for a whole body, and the response carries content_source: \"index\". Unknown or stale ids are listed in not_found and the remaining rows still return. Max 50 ids per call. Treat vector_id as a handle for a row you just found, not a durable identifier: the id scheme has been migrated before and may be again, so do not store one for later use.", + "maxItems": 50, + "type": "array", + "items": { + "type": "string", + "maxLength": 256 + } + }, + "graph_expand": { + "default": false, + "description": "Opt-in GraphRAG expansion (search mode only). When true, after fusion the top results seed a traversal of the Decision-Structure mention graph (D1 doc_edges); related wiki pages are returned in a separate graph_results array marked with graph_hop / graph_from — never mixed into results, and carrying no score. Default false preserves standard hybrid ranking (no graph read, no graph_results field).", + "type": "boolean" + }, + "use_memory": { + "default": false, + "description": "Order graph candidates within equal hops by your learned strength; keyword ranks and scores unchanged.", + "type": "boolean" + }, + "graph_hops": { + "default": 1, + "description": "Graph traversal depth for graph_expand (1 or 2). Default 1. Ignored when graph_expand is false.", + "type": "integer", + "minimum": 1, + "maximum": 2 + } + } + }, + "outputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "count": { + "type": "integer", + "minimum": 0, + "maximum": 9007199254740991 + }, + "mode": { + "type": "string", + "enum": [ + "search", + "scan", + "fetch" + ] + }, + "results": { + "maxItems": 50, + "type": "array", + "items": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + } + }, + "graph_results": { + "maxItems": 30, + "type": "array", + "items": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + } + }, + "trace_id": { + "type": "string" + }, + "timestamp": { + "type": "string" + }, + "memory_policy": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + }, + "memory_unavailable": { + "type": "boolean" + }, + "feedback_available": { + "type": "boolean" + } + }, + "required": [ + "count", + "mode", + "results", + "feedback_available" + ], + "additionalProperties": {} + } +} diff --git a/mcp-server/server/tools.d.ts b/mcp-server/server/tools.d.ts new file mode 100644 index 0000000..1f142f4 --- /dev/null +++ b/mcp-server/server/tools.d.ts @@ -0,0 +1,2 @@ +export const TOOLS: any[]; +export const MEMORY_CONTRACT: any; diff --git a/mcp-server/server/tools.js b/mcp-server/server/tools.js index fa954d1..6005a6b 100644 --- a/mcp-server/server/tools.js +++ b/mcp-server/server/tools.js @@ -1,3 +1,6 @@ +import { readFileSync } from "node:fs"; +const SEARCH_CONTRACT = JSON.parse(readFileSync(new URL("./search-schema.json", import.meta.url), "utf8")); +export const MEMORY_CONTRACT = JSON.parse(readFileSync(new URL("./memory-tools.json", import.meta.url), "utf8")); /** * Static mirror of the Worker `search` tool schema. * @@ -15,7 +18,7 @@ export const TOOLS = [ name: "search", title: "Search GitHub", description: - "Unified search across GitHub issues, PRs, releases, repository documentation, " + + MEMORY_CONTRACT.instructions + "\n" + "Unified search across GitHub issues, PRs, releases, repository documentation, " + "GitHub Wiki pages, commit diffs, issue/PR top-level comments, PR reviews, and " + "PR inline review comments. Four modes, all derived from the parameter set: " + "(1) hybrid semantic search — dense BGE-M3 + sparse BM25 over D1 FTS5 fused via RRF, then re-scored " + @@ -44,162 +47,13 @@ export const TOOLS = [ "absence of a score is not a score of zero). Triage: appearing on BOTH axes is the strongest signal — " + "two independent paths agreed. Keyword axis only = the words matched. Relationship axis only = the " + "vocabulary did not match but the entry is structurally adjacent to what did.", - inputSchema: { - type: "object", - properties: { - query: { - type: "string", - description: - "Natural language search query. Omit or leave empty to switch to metadata-only scan mode " + - "(results ordered by the timestamp implied by sort; default sort=\"updated_desc\" when empty).", - }, - repo: { - type: "string", - description: - "Filter by repository — full slug (owner/repo), exact match. " + - "A bare repository name (\"my-repo\") matches nothing and yields an empty result set; " + - "search mode flags that case as \"repo\" in the response's filters_unmatched.", - }, - path_prefix: { - type: "string", - description: - "Filter repository docs by a repository-relative directory prefix before ranking. " + - "Requires type=\"doc\", a trailing /, no leading /, backslash, NUL, empty, . or .. path segments, " + - "and at most 64 UTF-8 bytes. Search mode reports an unmatched value as \"path_prefix\" in filters_unmatched.", - }, - state: { - type: "string", - enum: ["open", "closed", "all"], - description: "Filter by state (default: all)", - }, - labels: { - type: "array", - items: { type: "string" }, - description: "Filter by label names (AND logic)", - }, - milestone: { - type: "string", - description: "Filter by milestone title", - }, - assignee: { - type: "string", - description: "Filter by assignee login", - }, - type: { - type: "string", - enum: [ - "issue", - "pull_request", - "release", - "doc", - "wiki_doc", - "diff", - "issue_comment", - "pr_review", - "pr_review_comment", - "all", - ], - description: - "Filter by type (default: all). " + - "\"doc\" = repository docs (files in /docs/ etc.). " + - "\"wiki_doc\" = GitHub Wiki pages (separate from repo docs; both surfaces co-exist). " + - "\"diff\" = per-file commit diffs. " + - "\"issue_comment\" = top-level comments on issues and PRs. " + - "\"pr_review\" = PR review bodies (approve / request_changes / comment). " + - "\"pr_review_comment\" = inline per-line review comments on PR diffs.", - }, - top_k: { - type: "number", - description: - "Max results (default: 10, max: 50). Counts distinct entities, not index rows " + - "(a file's doc row and its commit diffs collapse into one result).", - }, - fusion: { - type: "string", - enum: ["rrf", "dense_only", "sparse_only"], - description: - "Fusion strategy (default: rrf). dense_only / sparse_only for debugging or single-ranker queries. " + - "Ignored in scan mode (empty query).", - }, - rerank: { - type: "boolean", - description: - "Cross-encoder reranking with @cf/baai/bge-reranker-base (default: true). " + - "Set false to skip — faster, no rerank cost; recommended for short identifier queries or debugging. " + - "Ignored in scan mode (empty query).", - }, - sort: { - type: "string", - enum: ["relevance", "updated_desc", "created_desc"], - description: - "Result ordering. Default: \"relevance\" when query is non-empty, \"updated_desc\" when query is empty. " + - "Setting \"updated_desc\" / \"created_desc\" forces time-ordered output and overrides ranker scores.", - }, - since: { - type: "string", - description: - "ISO 8601 timestamp (inclusive) — keep only results whose updated_at >= since. " + - "Pair with sort=\"updated_desc\" + empty query for an activity scan. " + - "Default in scan mode: 7 days back from until (or from now when until is omitted).", - }, - until: { - type: "string", - description: - "ISO 8601 timestamp (exclusive) — keep only results whose updated_at < until. " + - "In scan mode the [since, until) window is applied inside the index, so any window " + - "holding rows returns rows however far back it sits; the response carries " + - "truncated: true when the window holds more than one page.", - }, - include_content: { - type: "boolean", - description: - "When true, inline the raw content of top doc and wiki_doc rows (docs via the GitHub " + - "contents API, wiki pages via raw.githubusercontent.com/wiki) on those rows. Capped at " + - "the first few rows of each surface to bound API fan-out. Other rows are unaffected. " + - "Default: false.", - }, - vector_ids: { - type: "array", - items: { type: "string" }, - maxItems: 50, - description: - "Stored-content fetch. Pass the vector_id values carried by earlier search-mode results " + - "(scan-mode rows come from the structured store and carry none) to read back the " + - "body text the index holds for those exact rows, for every type — issue, pull_request, " + - "issue_comment, pr_review, pr_review_comment, release, diff, doc, wiki_doc. " + - "Served from D1: no GitHub API call is made. Takes precedence over the other modes — query, sort, " + - "and every metadata filter are ignored when this is present, because the rows are named rather " + - "than selected. " + - "The text is the INDEXED copy of the body (the embedding input), truncated at 8000 characters — " + - "not the live source. Each row carries content_truncated so a prefix is never mistaken for a whole " + - "body, and the response carries content_source: \"index\". Unknown or stale ids are listed in " + - "not_found and the remaining rows still return. Max 50 ids per call. " + - "Treat vector_id as a handle for a row you just found, not a durable identifier: the id scheme has " + - "been migrated before and may be again, so do not store one for later use.", - }, - graph_expand: { - type: "boolean", - description: - "Opt-in GraphRAG expansion (search mode only). When true, after fusion the top " + - "results seed a traversal of the Decision-Structure mention graph (D1 doc_edges); " + - "related wiki pages are returned in a separate graph_results array marked with " + - "graph_hop / graph_from — never mixed into results, and carrying no score. " + - "Default false = byte-identical to standard hybrid retrieval (no graph read, " + - "no graph_results field).", - }, - graph_hops: { - type: "number", - minimum: 1, - maximum: 2, - description: - "Graph traversal depth for graph_expand (1 or 2). Default 1. " + - "Ignored when graph_expand is false.", - }, - }, - }, + inputSchema: SEARCH_CONTRACT.inputSchema, + outputSchema: SEARCH_CONTRACT.outputSchema, annotations: { - title: "Search GitHub", - readOnlyHint: true, + readOnlyHint: false, + destructiveHint: false, + openWorldHint: true, }, }, + ...MEMORY_CONTRACT.tools, ]; diff --git a/migrations/0008_source_event_identity.sql b/migrations/0008_source_event_identity.sql new file mode 100644 index 0000000..a4f2a15 --- /dev/null +++ b/migrations/0008_source_event_identity.sql @@ -0,0 +1,4 @@ +-- Add canonical event identities for sparse/fetch memory provenance. +-- Legacy comment/review rows must be reingested from GitHub before feedback is available. +ALTER TABLE search_docs ADD COLUMN comment_id INTEGER NOT NULL DEFAULT 0; +ALTER TABLE search_docs ADD COLUMN review_id INTEGER NOT NULL DEFAULT 0; diff --git a/scripts/check-schema-drift.mjs b/scripts/check-schema-drift.mjs index 8573fd5..39b54c7 100644 --- a/scripts/check-schema-drift.mjs +++ b/scripts/check-schema-drift.mjs @@ -1,174 +1,6 @@ #!/usr/bin/env node -// Guards against drift between the Worker `search` tool param schema and the -// client proxy's static mirror of it. -// -// source of truth : src/mcp.ts (inputSchema passed to server.registerTool("search", ...)) -// mirror : mcp-server/server/tools.js (TOOLS[0].inputSchema.properties) -// -// Why this exists (gh#157 / gh#159): the proxy answers tools/list from a -// hand-maintained static schema (it does NOT forward to the Worker, to keep -// startup auth-free / network-free). A param added to the Worker but forgotten -// in the proxy is silently stripped by MCP clients (additionalProperties:false) -// and never reaches the Worker — exactly how graph_expand shipped broken in -// v0.9.0. This check turns that "forgot to sync" procedure into a CI gate: the -// build fails instead of shipping a stale schema. -// -// Two axes are compared (gh#181): param NAMES and, for enum params, their -// VALUES. Name-only comparison let `type: "wiki_doc"` exist on the Worker while -// the proxy enum omitted it — clients validate against the proxy schema, so the -// value was rejected before any request left the client. - -import { readFileSync } from "node:fs"; +// Compare canonical schemas and actual served schemas with every shipped bridge tool. +import { execFileSync } from "node:child_process"; import { fileURLToPath } from "node:url"; -import { dirname, join } from "node:path"; - -const repoRoot = join(dirname(fileURLToPath(import.meta.url)), ".."); -const workerPath = join(repoRoot, "src", "mcp.ts"); -const proxyPath = join(repoRoot, "mcp-server", "server", "tools.js"); - -/** Return the substring of `text` from the first `startMarker` to the next `endMarker`. */ -function region(text, startMarker, endMarker, label) { - const start = text.indexOf(startMarker); - if (start === -1) throw new Error(`drift-check: start marker not found (${label}): ${startMarker}`); - const end = text.indexOf(endMarker, start + startMarker.length); - if (end === -1) throw new Error(`drift-check: end marker not found (${label}): ${endMarker}`); - return text.slice(start, end); -} - -/** Extract the string literals of the first `[ ... ]` list in `block`, if any. */ -function enumValues(block) { - const open = block.indexOf("["); - if (open === -1) return null; - const close = block.indexOf("]", open); - if (close === -1) return null; - const list = block.slice(open + 1, close); - const values = [...list.matchAll(/"([^"]*)"/g)].map((m) => m[1]); - return values.length > 0 ? values : null; -} - -/** - * Split a schema region into per-param blocks keyed by param name. - * `headerRe` must be a global, multiline regex whose first capture group is the - * param name; each block runs from its header to the next header (or the end). - */ -function paramBlocks(schema, headerRe) { - const heads = [...schema.matchAll(headerRe)]; - const blocks = new Map(); - for (let i = 0; i < heads.length; i++) { - const from = heads[i].index; - const to = i + 1 < heads.length ? heads[i + 1].index : schema.length; - blocks.set(heads[i][1], schema.slice(from, to)); - } - return blocks; -} - -// Worker: the zod object is `inputSchema` inside the config arg of -// server.registerTool("search", { description, inputSchema }, handler). -// Top-level param keys are 8-space-indented `: z`. Bounded from the tool -// name to the `async ({` handler so other zod objects in the file are excluded. -// Enum values come from the `.enum([...])` call inside each param's block. -function workerParams(text) { - const toolIdx = text.indexOf('"search"'); - if (toolIdx === -1) throw new Error("drift-check: `\"search\"` tool not found in Worker (src/mcp.ts)"); - const schema = region(text.slice(toolIdx), "{", "async ({", "worker search schema"); - const blocks = paramBlocks(schema, /^ {8}(\w+): z\b/gm); - const enums = new Map(); - for (const [name, block] of blocks) { - const enumIdx = block.indexOf(".enum("); - if (enumIdx === -1) continue; - const values = enumValues(block.slice(enumIdx)); - if (values) enums.set(name, values); - } - return { names: new Set(blocks.keys()), enums }; -} - -// Proxy: TOOLS[0].inputSchema.properties — top-level keys are 8-space-indented -// `: {`. Bounded from `properties: {` to the sibling `annotations:`. -// Enum values come from the `enum: [...]` key inside each param's block. -function proxyParams(text) { - const props = region(text, "properties: {", "annotations:", "proxy search schema"); - const blocks = paramBlocks(props, /^ {8}(\w+): \{/gm); - const enums = new Map(); - for (const [name, block] of blocks) { - const enumIdx = block.indexOf("enum: ["); - if (enumIdx === -1) continue; - const values = enumValues(block.slice(enumIdx)); - if (values) enums.set(name, values); - } - return { names: new Set(blocks.keys()), enums }; -} - -const worker = workerParams(readFileSync(workerPath, "utf8")); -const proxy = proxyParams(readFileSync(proxyPath, "utf8")); - -// Extractor sanity guard: a structural change could make a regex match nothing, -// turning the comparison into a meaningless empty==empty pass. Refuse that. -// The enum floor is the same guard on the value axis — `type` is the param this -// check exists for, and a zero-enum extraction would silently pass every value. -for (const [label, side] of [["worker", worker], ["proxy", proxy]]) { - if (side.names.size < 5 || !side.names.has("query") || !side.names.has("repo")) { - console.error( - `drift-check: ${label} param extraction looks wrong ` + - `(got ${side.names.size}: ${[...side.names].join(", ") || ""}). ` + - "The source structure likely changed — update scripts/check-schema-drift.mjs.", - ); - process.exit(2); - } - if (!side.enums.has("type") || side.enums.get("type").length < 5) { - console.error( - `drift-check: ${label} enum extraction looks wrong ` + - `(type = ${JSON.stringify(side.enums.get("type") ?? null)}). ` + - "The source structure likely changed — update scripts/check-schema-drift.mjs.", - ); - process.exit(2); - } -} - -const problems = []; - -const missingInProxy = [...worker.names].filter((k) => !proxy.names.has(k)); -const extraInProxy = [...proxy.names].filter((k) => !worker.names.has(k)); - -if (missingInProxy.length) - problems.push( - " missing in proxy (add to mcp-server/server/tools.js TOOLS search inputSchema.properties): " + - missingInProxy.join(", "), - ); -if (extraInProxy.length) - problems.push( - " extra in proxy (not present in Worker src/mcp.ts search schema): " + extraInProxy.join(", "), - ); - -// Enum values: compared as sets, per param present on both sides. A param whose -// enum exists on one side only is a drift too — the client would either reject a -// value the Worker accepts, or accept one it rejects. -for (const name of [...worker.names].filter((k) => proxy.names.has(k))) { - const w = worker.enums.get(name); - const p = proxy.enums.get(name); - if (!w && !p) continue; - if (!w || !p) { - problems.push( - ` enum mismatch on "${name}": ${w ? "Worker" : "proxy"} declares an enum, ` + - `${w ? "proxy" : "Worker"} does not.`, - ); - continue; - } - const missing = w.filter((v) => !p.includes(v)); - const extra = p.filter((v) => !w.includes(v)); - if (missing.length) - problems.push(` enum values missing in proxy for "${name}": ${missing.join(", ")}`); - if (extra.length) - problems.push(` enum values extra in proxy for "${name}": ${extra.join(", ")}`); -} - -if (problems.length) { - console.error("drift-check: proxy search schema is out of sync with the Worker."); - for (const line of problems) console.error(line); - process.exit(1); -} - -console.log( - `drift-check OK: proxy mirrors Worker search params (${worker.names.size}) ` + - `and enum values (${[...worker.enums.keys()].sort().join(", ")}): ` + - `${[...worker.names].sort().join(", ")}`, -); +const repoRoot = fileURLToPath(new URL("../", import.meta.url)); +execFileSync(process.execPath, ["node_modules/vitest/vitest.mjs", "run", "src/memory-contract.test.ts"], {cwd: repoRoot, stdio: "inherit"}); diff --git a/scripts/generate-tool-contracts.mjs b/scripts/generate-tool-contracts.mjs new file mode 100644 index 0000000..5b77b05 --- /dev/null +++ b/scripts/generate-tool-contracts.mjs @@ -0,0 +1,15 @@ +#!/usr/bin/env node +// Regenerate bridge JSON from the exact Worker Zod contracts. No Worker bindings. +import { readFileSync, writeFileSync } from 'node:fs'; +import { transpileModule, ModuleKind, ScriptTarget } from 'typescript'; +import { z } from 'zod'; +async function load(name) { + const source = readFileSync(new URL(`../src/${name}.ts`, import.meta.url), 'utf8'); + const js = transpileModule(source, { compilerOptions: { module: ModuleKind.ES2022, target: ScriptTarget.ES2022 } }).outputText.replace(/from ["']zod["']/g, `from ${JSON.stringify(import.meta.resolve('zod'))}`); + return import('data:text/javascript;base64,' + Buffer.from(js).toString('base64')); +} +const memory = await load('memory-contract'); +const search = await load('search-contract'); +const save = (name, data) => writeFileSync(new URL(`../mcp-server/server/${name}.json`, import.meta.url), JSON.stringify(data, null, 2) + '\n'); +save('memory-tools', { instructions: memory.MEMORY_INSTRUCTIONS, tools: Object.entries(memory.memorySchemas).map(([name, schema]) => ({ name, description: memory.memoryDescriptions[name], inputSchema: z.toJSONSchema(schema, { io: 'input' }), outputSchema: z.toJSONSchema(memory.memoryResultSchema), annotations: { readOnlyHint: name === 'memory_history', destructiveHint: false, idempotentHint: true, openWorldHint: false } })) }); +save('search-schema', { inputSchema: z.toJSONSchema(search.searchInputSchema, { io: 'input' }), outputSchema: z.toJSONSchema(search.searchOutputSchema) }); diff --git a/src/backfill-source-identities.ts b/src/backfill-source-identities.ts new file mode 100644 index 0000000..c5493f6 --- /dev/null +++ b/src/backfill-source-identities.ts @@ -0,0 +1,16 @@ +import { issueCommentVectorId, prReviewVectorId, prReviewCommentVectorId } from './pipeline/vector-id.js'; +export async function backfillSourceIdentities(db: D1Database, store: { fetch(request: Request): Promise }, opts: { repo: string; cursor?: string; limit?: number }) { + const limit = opts.limit ?? 50; + if (!opts.repo || opts.repo.length > 256 || !Number.isInteger(limit) || limit < 1 || limit > 100 || (opts.cursor && !/^(issue_comment|pr_review|pr_review_comment):[0-9]+$/.test(opts.cursor))) throw new Error('Invalid identity backfill options'); + const url = new URL('http://store/source-identities'); url.searchParams.set('repo', opts.repo); url.searchParams.set('limit', String(limit)); if (opts.cursor) url.searchParams.set('cursor', opts.cursor); + const res = await store.fetch(new Request(url)); if (!res.ok) throw new Error('Identity page unavailable'); + const page = await res.json() as { rows: { repo: string; type: string; event_id: number }[]; next_cursor: string | null; done: boolean }; + const statements: D1PreparedStatement[] = []; + for (const r of page.rows) { + const id = r.type === 'issue_comment' ? await issueCommentVectorId(r.repo, r.event_id) : r.type === 'pr_review' ? await prReviewVectorId(r.repo, r.event_id) : await prReviewCommentVectorId(r.repo, r.event_id); + const column = r.type === 'pr_review' ? 'review_id' : 'comment_id'; + statements.push(db.prepare(`UPDATE search_docs SET ${column}=? WHERE vector_id=? AND repo=? AND type=? AND ${column}<>? RETURNING vector_id`).bind(r.event_id, id, r.repo, r.type, r.event_id)); + } + const result = statements.length ? await db.batch(statements) : []; + return { scanned: page.rows.length, updated: result.reduce((n,r) => n + (r.results?.length ?? 0), 0), next_cursor: page.next_cursor, done: page.done }; +} diff --git a/src/fetch.ts b/src/fetch.ts index cb80e8b..83b7c6b 100644 --- a/src/fetch.ts +++ b/src/fetch.ts @@ -69,6 +69,8 @@ export interface FetchItem { * a prefix as whole. */ content_truncated: boolean; + comment_id?: number; + review_id?: number; tag_name?: string; doc_path?: string; wiki_path?: string; @@ -124,6 +126,8 @@ export function buildFetchItem( content_chars: content.length, content_truncated: content.length >= MAX_EMBEDDING_INPUT_CHARS, }; + if (type === "issue_comment" || type === "pr_review_comment") item.comment_id = Number(row.comment_id ?? 0); + if (type === "pr_review") item.review_id = Number(row.review_id ?? 0); if (type === "release") item.tag_name = String(row.tag_name ?? ""); if (type === "doc") item.doc_path = path; if (type === "wiki_doc") item.wiki_path = path; diff --git a/src/fts.ts b/src/fts.ts index 82ae600..f5f6710 100644 --- a/src/fts.ts +++ b/src/fts.ts @@ -62,6 +62,8 @@ export interface FtsUpsertRow { fileStatus?: DiffFileStatus | ""; commitDate?: string; commitAuthor?: string; + commentId?: number; + reviewId?: number; content: string; // tokenizable text (title+body or commit msg + path + patch) } @@ -120,8 +122,8 @@ export async function upsertFtsRow( `INSERT INTO search_docs ( vector_id, repo, type, state, labels, milestone, assignees, updated_at, number, tag_name, doc_path, commit_sha, file_path, file_status, - commit_date, commit_author, tokenizer_kind, content, content_fts, indexed_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + commit_date, commit_author, comment_id, review_id, tokenizer_kind, content, content_fts, indexed_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT (vector_id) DO UPDATE SET repo = excluded.repo, type = excluded.type, @@ -138,6 +140,8 @@ export async function upsertFtsRow( file_status = excluded.file_status, commit_date = excluded.commit_date, commit_author = excluded.commit_author, + comment_id = excluded.comment_id, + review_id = excluded.review_id, tokenizer_kind = excluded.tokenizer_kind, content = excluded.content, content_fts = excluded.content_fts, @@ -160,6 +164,8 @@ export async function upsertFtsRow( row.fileStatus ?? "", row.commitDate ?? "", row.commitAuthor ?? "", + row.commentId ?? 0, + row.reviewId ?? 0, tokenizerKind, row.content, ftsIndexText(row.content, tokenizerKind), @@ -326,10 +332,7 @@ export async function detectUnmatchedFilters( if (!repoMatched) unmatched.push("repo"); } catch (err) { repoMatched = false; - console.error( - "detectUnmatchedFilters: repo probe failed:", - err instanceof Error ? err.message : String(err), - ); + console.error("detectUnmatchedFilters: repo probe failed:"); } } if (filters.pathPrefix && repoMatched) { @@ -338,10 +341,7 @@ export async function detectUnmatchedFilters( unmatched.push("path_prefix"); } } catch (err) { - console.error( - "detectUnmatchedFilters: path_prefix probe failed:", - err instanceof Error ? err.message : String(err), - ); + console.error("detectUnmatchedFilters: path_prefix probe failed:"); } } return unmatched; @@ -365,6 +365,8 @@ export interface FtsHit { fileStatus: string; commitDate: string; commitAuthor: string; + commentId?: number; + reviewId?: number; content: string; score: number; } @@ -483,7 +485,7 @@ export async function queryFts( SELECT d.vector_id AS vector_id, d.repo, d.type, d.state, d.labels, d.milestone, d.assignees, d.updated_at, d.number, d.tag_name, d.doc_path, d.commit_sha, d.file_path, d.file_status, - d.commit_date, d.commit_author, d.content, + d.commit_date, d.commit_author, d.comment_id, d.review_id, d.content, bm25(search_docs_nat_fts_v3) AS score, ${tier} AS tier FROM search_docs_nat_fts_v3 f JOIN search_docs d ON d.rowid = f.rowid @@ -497,7 +499,7 @@ export async function queryFts( SELECT d.vector_id AS vector_id, d.repo, d.type, d.state, d.labels, d.milestone, d.assignees, d.updated_at, d.number, d.tag_name, d.doc_path, d.commit_sha, d.file_path, d.file_status, - d.commit_date, d.commit_author, d.content, + d.commit_date, d.commit_author, d.comment_id, d.review_id, d.content, bm25(search_docs_code_fts_v2) AS score, 0 AS tier FROM search_docs_code_fts_v2 f JOIN search_docs d ON d.rowid = f.rowid @@ -562,6 +564,8 @@ export async function queryFts( fileStatus: String(r.file_status ?? ""), commitDate: String(r.commit_date ?? ""), commitAuthor: String(r.commit_author ?? ""), + commentId: Number(r.comment_id ?? 0), + reviewId: Number(r.review_id ?? 0), content: String(r.content ?? ""), score: Number(r.score ?? 0), }); @@ -645,3 +649,8 @@ export function reciprocalRankFusion( contributions: contributions.get(vectorId) ?? {}, })); } + +/** Repair canonical event IDs even when unchanged content skips embedding. */ +export async function repairSourceEventIdentity(db: D1Database, vectorId: string, kind: 'comment' | 'review', id: number): Promise { + await db.prepare(`UPDATE search_docs SET ${kind === 'comment' ? 'comment_id' : 'review_id'}=? WHERE vector_id=?`).bind(id, vectorId).run(); +} diff --git a/src/fts.workers.test.ts b/src/fts.workers.test.ts index ca432de..1730dd9 100644 --- a/src/fts.workers.test.ts +++ b/src/fts.workers.test.ts @@ -215,7 +215,7 @@ describe("fts D1: v1 corruption recovery migration", () => { }), ).rejects.toThrow(/SQLITE_CORRUPT_VTAB/); - await applyD1Migrations(env.DB_FTS_MIGRATION, [repairMigration!, segmentMigration!]); + await applyD1Migrations(env.DB_FTS_MIGRATION, [repairMigration!, segmentMigration!, env.TEST_MIGRATIONS.find(m => m.name === "0008_source_event_identity.sql")!]); await env.DB_FTS_MIGRATION .prepare( diff --git a/src/graph.ts b/src/graph.ts index 4113fee..a730fc2 100644 --- a/src/graph.ts +++ b/src/graph.ts @@ -30,6 +30,7 @@ export interface GraphNeighbor { vectorId: string; hop: number; fromVectorId: string; + path?: import("./memory.js").SavedEdge[]; } /** Escape a string for safe inclusion in a RegExp. */ @@ -165,23 +166,24 @@ export async function queryNeighbors( const limit = Math.max(1, Math.min(200, opts.limit ?? 50)); // Anchor: each seed at depth 0 with itself as origin. - const seedValues = seeds.map(() => "(?, 0, ?)").join(", "); + const seedValues = seeds.map(() => "(?, 0, ?, json_array())").join(", "); const repoFilter = opts.repo ? "AND e.repo = ?" : ""; const sql = ` - WITH RECURSIVE reach(id, depth, origin) AS ( + WITH RECURSIVE reach(id, depth, origin, path) AS ( SELECT * FROM (VALUES ${seedValues}) UNION SELECT CASE WHEN e.src_vector_id = r.id THEN e.dst_vector_id ELSE e.src_vector_id END, r.depth + 1, - r.origin + r.origin, + json_insert(r.path, '$[#]', json_object('repo', e.repo, 'src', e.src_slug, 'dst', e.dst_slug, 'kind', e.edge_kind)) FROM doc_edges e JOIN reach r ON (e.src_vector_id = r.id OR e.dst_vector_id = r.id) WHERE r.depth < ? ${repoFilter} ) - SELECT id, MIN(depth) AS hop, origin + SELECT id, MIN(depth) AS hop, origin, path FROM reach WHERE depth > 0 GROUP BY id @@ -198,7 +200,7 @@ export async function queryNeighbors( const res = await db .prepare(sql) .bind(...binds) - .all<{ id: string; hop: number; origin: string }>(); + .all<{ id: string; hop: number; origin: string; path: string }>(); const seedSet = new Set(seeds); return (res.results ?? []) @@ -206,6 +208,7 @@ export async function queryNeighbors( vectorId: String(r.id ?? ""), hop: Number(r.hop ?? 0), fromVectorId: String(r.origin ?? ""), + path: JSON.parse(r.path), })) .filter((n) => n.vectorId.length > 0 && !seedSet.has(n.vectorId)); } @@ -231,7 +234,7 @@ export async function getDocsByVectorIds( .prepare( `SELECT vector_id, repo, type, state, labels, milestone, assignees, updated_at, number, tag_name, doc_path, commit_sha, file_path, file_status, - commit_date, commit_author, tokenizer_kind, content, indexed_at + commit_date, commit_author, comment_id, review_id, tokenizer_kind, content, indexed_at FROM search_docs WHERE vector_id IN (${placeholders})`, ) diff --git a/src/index.ts b/src/index.ts index cc69e32..008bf01 100644 --- a/src/index.ts +++ b/src/index.ts @@ -30,6 +30,7 @@ * Hourly (fallback) -- poll GitHub API for issue/PR updates, generate embeddings, upsert vectors */ +import { backfillSourceIdentities } from "./backfill-source-identities.js"; import type { Env } from "./types.js"; import { createOAuthProvider, @@ -118,6 +119,15 @@ const innerHandler: ExportedHandler = { return handleWebhook(request, env); } + // Bounded repair from existing canonical DO event IDs; no GitHub fetch or embedding. + if (request.method === "POST" && url.pathname === "/admin/backfill-source-identities") { + if (!env.GITHUB_TOKEN || request.headers.get("GITHUB_TOKEN") !== env.GITHUB_TOKEN) return new Response("Unauthorized", { status: 401 }); + try { + const result = await backfillSourceIdentities(env.DB_FTS, env.ISSUE_STORE.get(env.ISSUE_STORE.idFromName("global")), { repo: url.searchParams.get("repo") ?? "", cursor: url.searchParams.get("cursor") ?? undefined, limit: Number(url.searchParams.get("limit") ?? 50) }); + return Response.json(result); + } catch { return Response.json({ error: "Identity backfill failed; cursor unchanged" }, { status: 400 }); } + } + // -- Admin: reset hashes and watermarks to trigger full re-embedding on next cron -- // POST /admin/reset-hashes?repo=owner/repo // Resets: issue body_hash, release body_hash, docs (deleted), watermarks, and diff --git a/src/mcp-stateless-contract.test.ts b/src/mcp-stateless-contract.test.ts index 3272d5b..dba4722 100644 --- a/src/mcp-stateless-contract.test.ts +++ b/src/mcp-stateless-contract.test.ts @@ -83,7 +83,7 @@ describe("worker <-> bridge stateless contract", () => { const client = await remote.getClient(); const tools = await client.listTools(); - expect(tools.tools.map((t: { name: string }) => t.name)).toEqual(["search"]); + expect(tools.tools.map((t: { name: string }) => t.name)).toEqual(["memory_history", "record_source_use", "record_outcome", "search"]); // The whole exchange, connect included, is POST-only and session-free. expect(seen.length).toBeGreaterThan(0); @@ -116,7 +116,7 @@ describe("worker <-> bridge stateless contract", () => { }); const client = await remote.getClient(); - const [search] = (await client.listTools()).tools; + const search = (await client.listTools()).tools.find((t: any) => t.name === "search")!; // The drift check compares source text; this asserts the schema actually // survives registration and serialization to the client. @@ -141,6 +141,7 @@ describe("worker <-> bridge stateless contract", () => { "type", "until", "vector_ids", + "use_memory", ].sort(), ); @@ -159,7 +160,7 @@ describe("worker <-> bridge stateless contract", () => { }); const client = await remote.getClient(); - const [search] = (await client.listTools()).tools; + const search = (await client.listTools()).tools.find((t: any) => t.name === "search")!; const description = search.description as string; expect(description).toContain("vector_ids"); @@ -188,7 +189,7 @@ describe("worker <-> bridge stateless contract", () => { }); const client = await remote.getClient(); - const [search] = (await client.listTools()).tools; + const search = (await client.listTools()).tools.find((t: any) => t.name === "search")!; const description = search.description as string; expect(description).toContain("graph_results"); diff --git a/src/mcp.ts b/src/mcp.ts index 59ad7d1..ecd00d3 100644 --- a/src/mcp.ts +++ b/src/mcp.ts @@ -1,5 +1,5 @@ /** - * MCP server factory exposing a single consolidated semantic search tool. + * MCP server factory exposing consolidated retrieval and private feedback tools. * * Tools: * search — hybrid search + time-ordered activity scan + inline doc @@ -22,7 +22,7 @@ * * Protocol revision 2026-07-28 (stateless core, issue #224). The server is * built fresh per HTTP request by `createMcpHandler` in `index.ts` — there is - * no session, no `initialize`, and no Durable Object in the serving path. What + * no session or `initialize`; private memory uses the existing IssueStore DO. What * `McpAgent` used to provide in two roles is now split: instance resolution * from a session ID is gone outright, and user identity comes from the OAuth * props of the request being served, read through `getMcpAuthContext()`. @@ -32,9 +32,10 @@ * `retired-do.ts` so this module imports nothing from `cloudflare:workers`. */ +import { registerMemoryTools, rememberResult, memoryCall, hashReturnedContent, MEMORY_INSTRUCTIONS } from "./memory-api.js"; import { getMcpAuthContext } from "agents/mcp/server"; import { McpServer } from "@modelcontextprotocol/server"; -import { z } from "zod"; +import { searchInputSchema, searchOutputSchema, INCLUDE_CONTENT_MAX_DOCS } from "./search-contract.js"; import type { Env, IssueRecord, @@ -105,7 +106,7 @@ function githubHeaders(token: string): Record { * top_k for generic scanning. Callers that need more should page via * additional queries rather than lifting this cap. */ -const INCLUDE_CONTENT_MAX_DOCS = 5; + /** * Per-candidate payload assembled after fusion: the dense metadata, the sparse @@ -180,8 +181,8 @@ function resolveRow(p: RowPayload | undefined): ResolvedRow { commitDate: meta?.commit_date ?? ftsRow?.commitDate ?? "", commitAuthor: meta?.commit_author ?? ftsRow?.commitAuthor ?? "", author: meta?.author ?? "", - commentId: meta?.comment_id ?? 0, - reviewId: meta?.review_id ?? 0, + commentId: meta?.comment_id ?? ftsRow?.commentId ?? 0, + reviewId: meta?.review_id ?? ftsRow?.reviewId ?? 0, line: meta?.line ?? 0, }; } @@ -237,8 +238,11 @@ export type GraphItem = { wiki_path?: string; doc_path?: string; content?: string; + content_source?: string; graph_hop: number; graph_from: string; + graph_path?: import("./memory.js").SavedEdge[]; + learned_strength?: number; }; /** @@ -273,11 +277,13 @@ export function buildGraphItem( repo, graph_hop: neighbor.hop, graph_from: fromSlug, + graph_path: neighbor.path ?? [], }; if (type === "wiki_doc") item.wiki_path = path; if (type === "doc") item.doc_path = path; if (includeContent && typeof row.content === "string") { item.content = row.content; + item.content_source = "index"; } return item; } @@ -294,14 +300,16 @@ export function createRagMcpServer(env: Env): McpServer { const server = new McpServer({ name: "github-rag-mcp", version: "0.1.0", - }); + }, { instructions: MEMORY_INSTRUCTIONS }); + registerMemoryTools(server, env); // ── search ────────────────────────────────────────── server.registerTool( "search", { + annotations: { readOnlyHint: false, destructiveHint: false, openWorldHint: true }, description: - "Unified search across GitHub issues, PRs, releases, repository documentation, GitHub Wiki pages, " + + MEMORY_INSTRUCTIONS + "\n" + "Unified search across GitHub issues, PRs, releases, repository documentation, GitHub Wiki pages, " + "commit diffs, issue/PR top-level comments, PR reviews, and PR inline review comments. " + "Four modes, all derived from the parameter set:\n" + " 1. Hybrid semantic search (default): dense BGE-M3 over Vectorize + sparse BM25 over D1 FTS5, " + @@ -340,178 +348,11 @@ export function createRagMcpServer(env: Env): McpServer { "absence of a score is not a score of zero). Triage: appearing on BOTH axes is the strongest signal — " + "two independent paths agreed. Keyword axis only = the words matched. Relationship axis only = the " + "vocabulary did not match but the entry is structurally adjacent to what did.", - inputSchema: z.object({ - query: z - .string() - .optional() - .describe( - "Natural language search query. When omitted or empty, the tool " + - "switches to metadata-only scan mode and results are ordered " + - "by the timestamp implied by sort (default sort=\"updated_desc\" for empty query).", - ), - repo: z - .string() - .optional() - .describe( - "Filter by repository — full slug (owner/repo), exact match. " + - "A bare repository name (\"my-repo\") matches nothing and yields an empty result set; " + - "search mode flags that case as \"repo\" in the response's filters_unmatched.", - ), - path_prefix: z - .string() - .optional() - .describe( - "Filter repository docs by a repository-relative directory prefix before ranking. " + - "Requires type=\"doc\", a trailing /, no leading /, backslash, NUL, empty, . or .. path segments, " + - "and at most 64 UTF-8 bytes. Search mode reports an unmatched value as \"path_prefix\" in filters_unmatched.", - ), - state: z - .enum(["open", "closed", "all"]) - .optional() - .default("all") - .describe("Filter by state"), - labels: z - .array(z.string()) - .optional() - .describe("Filter by label names (AND logic)"), - milestone: z - .string() - .optional() - .describe("Filter by milestone title"), - assignee: z - .string() - .optional() - .describe("Filter by assignee login"), - type: z - .enum([ - "issue", - "pull_request", - "release", - "doc", - "wiki_doc", - "diff", - "issue_comment", - "pr_review", - "pr_review_comment", - "all", - ]) - .optional() - .default("all") - .describe( - "Filter by type (default: all). " + - "\"doc\" = repository docs (files in /docs/ etc.). " + - "\"wiki_doc\" = GitHub Wiki pages (separate from repo docs; both surfaces co-exist). " + - "\"diff\" = per-file commit diffs. " + - "\"issue_comment\" = top-level comments on issues and PRs. " + - "\"pr_review\" = PR review bodies (approve / request_changes / comment). " + - "\"pr_review_comment\" = inline per-line review comments on PR diffs.", - ), - top_k: z - .number() - .min(1) - .max(50) - .optional() - .default(10) - .describe("Max results (default: 10, max: 50)"), - fusion: z - .enum(["rrf", "dense_only", "sparse_only"]) - .optional() - .default("rrf") - .describe( - "Fusion strategy. Default: rrf (Reciprocal Rank Fusion over dense + sparse). " + - "dense_only = Vectorize only. sparse_only = D1 FTS5 BM25 only. " + - "Use rrf unless debugging a specific ranker. Ignored in metadata-only scan mode (empty query).", - ), - rerank: z - .boolean() - .optional() - .default(true) - .describe( - "Cross-encoder reranking with @cf/baai/bge-reranker-base. Default: true. " + - "When enabled, the fused (or single-ranker) candidates are overfetched (top_k × 5, max 50), " + - "post-filtered, then re-scored by the cross-encoder before being trimmed to top_k. " + - "Set false to disable (faster, no Workers AI rerank cost; recommended for debugging or " + - "when query is a short identifier where lexical match is already decisive). " + - "Ignored in metadata-only scan mode (empty query).", - ), - sort: z - .enum(["relevance", "updated_desc", "created_desc"]) - .optional() - .describe( - "Result ordering. Default: \"relevance\" when query is non-empty, \"updated_desc\" when query is empty. " + - "\"updated_desc\" / \"created_desc\" force time-ordered output and override ranker scores.", - ), - since: z - .string() - .optional() - .describe( - "ISO 8601 timestamp (inclusive) — keep only results whose updated_at >= since. " + - "Pair with sort=\"updated_desc\" + empty query for an activity scan. " + - "Default in scan mode: 7 days back from until (or from now when until is omitted).", - ), - until: z - .string() - .optional() - .describe( - "ISO 8601 timestamp (exclusive) — keep only results whose updated_at < until. " + - "In scan mode the [since, until) window is applied inside the index, so any window " + - "holding rows returns rows however far back it sits; the response carries " + - "truncated: true when the window holds more than one page.", - ), - include_content: z - .boolean() - .optional() - .default(false) - .describe( - "When true and a result row is type=\"doc\", fetch the file content from the GitHub " + - "contents API and inline it as a \"content\" field on that row. Capped at the first " + - `${INCLUDE_CONTENT_MAX_DOCS} doc rows in the result set to bound API fan-out. ` + - "Non-doc rows are unaffected.", - ), - vector_ids: z - .array(z.string()) - .max(FETCH_MAX_VECTOR_IDS) - .optional() - .describe( - "Stored-content fetch. Pass the vector_id values carried by earlier search-mode results " + - "(scan-mode rows come from the structured store and carry none) to read back the " + - "body text the index holds for those exact rows, for every type — issue, pull_request, " + - "issue_comment, pr_review, pr_review_comment, release, diff, doc, wiki_doc. " + - "Served from D1: no GitHub API call is made. Takes precedence over the other modes — query, " + - "sort, and every metadata filter are ignored when this is present, because the rows are named " + - "rather than selected. " + - "The text is the INDEXED copy of the body (the embedding input), truncated at " + - `${FETCH_CONTENT_MAX_CHARS} characters — not the live source. Each row carries content_truncated ` + - "so a prefix is never mistaken for a whole body, and the response carries content_source: \"index\". " + - "Unknown or stale ids are listed in not_found and the remaining rows still return. " + - `Max ${FETCH_MAX_VECTOR_IDS} ids per call. ` + - "Treat vector_id as a handle for a row you just found, not a durable identifier: the id scheme " + - "has been migrated before and may be again, so do not store one for later use.", - ), - graph_expand: z - .boolean() - .optional() - .default(false) - .describe( - "Opt-in GraphRAG expansion (search mode only). When true, after fusion the top " + - "results seed a traversal of the Decision-Structure mention graph (D1 doc_edges); " + - "related wiki pages are returned in a separate graph_results array marked with " + - "graph_hop / graph_from — never mixed into results, and carrying no score. " + - "Default false = byte-identical to standard hybrid retrieval (no graph read, " + - "no graph_results field).", - ), - graph_hops: z - .number() - .min(1) - .max(2) - .optional() - .default(1) - .describe( - "Graph traversal depth for graph_expand (1 or 2). Default 1. Ignored when graph_expand is false.", - ), - }), + inputSchema: searchInputSchema, + outputSchema: searchOutputSchema, }, - async ({ + async (args) => { + const result = await (async ({ query, repo, path_prefix, @@ -530,7 +371,11 @@ export function createRagMcpServer(env: Env): McpServer { vector_ids, graph_expand, graph_hops, + use_memory, }) => { + const memoryRequest = { query, repo, path_prefix, state, labels, milestone, assignee, type, top_k, fusion, rerank, sort, since, until, include_content, vector_ids, graph_expand, graph_hops, use_memory }; + const remember = (payload: unknown) => rememberResult(env, payload, memoryRequest); + // ── Fetch mode (vector_ids): stored content for named rows ─── // First branch on purpose: a fetch call carries no query, so leaving it // below the empty-query test would route it into scan mode. The rows are @@ -541,17 +386,14 @@ export function createRagMcpServer(env: Env): McpServer { const payload = await fetchStoredContent(env.DB_FTS, fetchIds); return { content: [ - { type: "text" as const, text: JSON.stringify(payload, null, 2) }, + { type: "text" as const, text: JSON.stringify(await remember(payload), null, 2) }, ], }; } catch (err) { // Reported as an error rather than as an all-missing response: an // empty `results` with every id in `not_found` would assert the rows // do not exist, which a failed read gives no ground to claim. - console.error( - "search: stored-content fetch failed:", - err instanceof Error ? err.message : String(err), - ); + console.error("search: stored-content fetch failed:"); return { content: [ { @@ -613,7 +455,7 @@ export function createRagMcpServer(env: Env): McpServer { { type: "text" as const, text: JSON.stringify( - { + await remember({ count: items.length, mode: "scan", sort: effectiveSort, @@ -621,7 +463,8 @@ export function createRagMcpServer(env: Env): McpServer { until: until ?? null, truncated: scan.truncated, results: items, - }, + memory_incomplete: scan.incomplete, + }), null, 2, ), @@ -693,6 +536,7 @@ export function createRagMcpServer(env: Env): McpServer { })(); // ── Sparse path: D1 FTS5 BM25 query ────────────────────── + let retrievalIncomplete = false; const sparsePromise: Promise = fusionMode === "dense_only" ? Promise.resolve([]) @@ -710,10 +554,8 @@ export function createRagMcpServer(env: Env): McpServer { try { return await queryFts(env.DB_FTS, trimmedQuery, internalTopK, ftsFilter); } catch (err) { - console.error( - "search: D1 FTS5 query failed:", - err instanceof Error ? err.message : String(err), - ); + retrievalIncomplete = true; + console.error("search: D1 FTS5 query failed:"); return []; } })(); @@ -910,10 +752,7 @@ export function createRagMcpServer(env: Env): McpServer { } catch (err) { // Backfill is best-effort: a D1 failure must not take the search // down. We fall through with whatever content we already have. - console.error( - "search: rerank content backfill failed:", - err instanceof Error ? err.message : String(err), - ); + console.error("search: rerank content backfill failed:"); } } @@ -1097,6 +936,7 @@ export function createRagMcpServer(env: Env): McpServer { const or = resolveRow(payload.get(o.vectorId)); return { vector_id: o.vectorId, + repo: or.repo, number: or.number, doc_path: or.docPath, wiki_path: or.wikiPath, file_path: or.filePath, tag_name: or.tagName, comment_id: or.commentId, review_id: or.reviewId, type: or.type, url: buildResultUrl(or), updated_at: or.updatedAt, @@ -1241,13 +1081,14 @@ export function createRagMcpServer(env: Env): McpServer { } // ── Optional graph expansion (opt-in; default off leaves everything - // above byte-identical). Seeds from the final result set, traverses the + // above unchanged in ranking). Seeds from the final result set, traverses the // Decision-Structure mention graph, and returns related wiki pages as a // SEPARATE axis (issue #234) — never mixed into `results`. Best-effort: // any failure returns the keyword axis unchanged with an empty graph // axis. `queryNeighbors` already orders by hop ascending and the skip // below only removes rows, so the axis stays hop-ordered. const graphItems: GraphItem[] = []; + let memoryIncomplete = retrievalIncomplete; if (graphExpand && filtered.length > 0) { try { const seedIds = filtered.map((f) => f.vectorId); @@ -1255,9 +1096,14 @@ export function createRagMcpServer(env: Env): McpServer { const neighbors = await queryNeighbors(env.DB_FTS, seedIds, { hops: graphHops, repo, - limit: Math.min(requestedTopK * 2, 30), + limit: use_memory ? 200 : Math.min(requestedTopK * 2, 30), }); - const fresh = neighbors.filter((n) => !seedSet.has(n.vectorId)); + if (use_memory) { + const strengths: number[] = await memoryCall(env, "strengths", neighbors.map(n => n.path ?? [])); + neighbors.forEach((n, i) => Object.assign(n, { learnedStrength: strengths[i] })); + neighbors.sort((a, b) => a.hop - b.hop || Number((b as any).learnedStrength) - Number((a as any).learnedStrength)); + } + const fresh = neighbors.filter((n) => !seedSet.has(n.vectorId)).slice(0, Math.min(requestedTopK * 2, 30)); if (fresh.length > 0) { const enrich = await getDocsByVectorIds( env.DB_FTS, @@ -1271,15 +1117,13 @@ export function createRagMcpServer(env: Env): McpServer { const row = enrich.get(n.vectorId); if (!row) continue; // dangling edge (target not indexed) — skip graphItems.push( - buildGraphItem(n, row, slugOf(n.fromVectorId), includeContent), + { ...buildGraphItem(n, row, slugOf(n.fromVectorId), includeContent), ...(use_memory ? { learned_strength: Number((n as any).learnedStrength) } : {}) }, ); } } } catch (graphErr) { - console.error( - "graph_expand failed (returning organic results):", - graphErr instanceof Error ? graphErr.message : String(graphErr), - ); + memoryIncomplete = true; + console.error("graph_expand failed (returning organic results):"); } } @@ -1288,10 +1132,11 @@ export function createRagMcpServer(env: Env): McpServer { { type: "text" as const, text: JSON.stringify( - { + await remember({ // Keyword-axis count only (issue #234). Relationship-axis // rows are counted by `graph_neighbors` and returned in // `graph_results`. + memory_incomplete: memoryIncomplete, count: items.length, mode: "search", fusion: fusionMode, @@ -1324,17 +1169,20 @@ export function createRagMcpServer(env: Env): McpServer { results: items, // Relationship axis (issue #234). Present only when // `graph_expand: true`, so the default response stays - // byte-identical to the pre-#234 shape. Ordered by + // free of the graph_results field. Ordered by // `graph_hop` ascending; carries no ranker score because the // axis has none to report. ...(graphExpand ? { graph_results: graphItems } : {}), - }, + }), null, 2, ), }, ], }; + })(args); + if (result.isError) return result; + return { ...result, structuredContent: JSON.parse(result.content[0].text) }; }, ); @@ -1364,6 +1212,8 @@ async function inlineDocContent< wiki_path?: string; wiki_extension?: string; content?: string; + content_source?: string; + content_version?: string; }, >(rows: T[], fallbackRepo?: string): Promise { const docRows = rows.filter((r) => r.type === "doc"); @@ -1389,10 +1239,12 @@ async function inlineDocContent< content?: string; encoding?: string; }; - if (!data.content) return; + if (typeof data.content !== "string" || data.encoding !== "base64") return; const binary = atob(data.content.replace(/\n/g, "")); const bytes = Uint8Array.from(binary, (c) => c.charCodeAt(0)); - row.content = new TextDecoder().decode(bytes); + const content = new TextDecoder().decode(bytes); + const content_version = await hashReturnedContent(content); + Object.assign(row, { content, content_source: "github_live", content_version }); } catch { // Best-effort inline; a failed fetch leaves `content` unset. } @@ -1408,7 +1260,9 @@ async function inlineDocContent< headers: { "User-Agent": "github-rag-mcp/0.1.0" }, }); if (!res.ok) return; - row.content = await res.text(); + const content = await res.text(); + const content_version = await hashReturnedContent(content); + Object.assign(row, { content, content_source: "github_live", content_version }); } catch { // Best-effort inline; a failed fetch leaves `content` unset. } diff --git a/src/memory-api.ts b/src/memory-api.ts new file mode 100644 index 0000000..9cad143 --- /dev/null +++ b/src/memory-api.ts @@ -0,0 +1,59 @@ +import type { SavedEdge, SavedSource } from './memory.js'; +import { MEMORY_INSTRUCTIONS, memoryDescriptions, memorySchemas, memoryResultSchema } from './memory-contract.js'; +import { getMcpAuthContext } from 'agents/mcp/server'; +import type { McpServer } from '@modelcontextprotocol/server'; +import type { Env } from './types.js'; +import type { McpProps } from './oauth.js'; +export async function memoryCall(env: Env, action: string, args: unknown): Promise { + const id = (getMcpAuthContext()?.props as McpProps | undefined)?.githubUserId; + if (!Number.isSafeInteger(id) || Number(id) <= 0) throw new Error('Authenticated principal required'); + const stub = env.ISSUE_STORE.get(env.ISSUE_STORE.idFromName('global')); + const res = await stub.fetch(new Request('http://store/memory', { method: 'POST', body: JSON.stringify({ principal: `github:${id}`, action, args }) })); + if (!res.ok) { const failure = await res.json() as { error?: string }; throw new Error(failure.error ?? 'invalid_memory_request'); } + return res.json(); +} +export function registerMemoryTools(server: McpServer, env: Env) { + for (const name of Object.keys(memorySchemas) as (keyof typeof memorySchemas)[]) { + server.registerTool(name, { description: memoryDescriptions[name], inputSchema: memorySchemas[name], outputSchema: memoryResultSchema, + annotations: { readOnlyHint: name === 'memory_history', destructiveHint: false, idempotentHint: true, openWorldHint: false } }, async (args: unknown) => { + try { const data = await memoryCall(env, name, args); return { content: [{ type: 'text' as const, text: JSON.stringify({ data }) }], structuredContent: { data } }; } + catch (err) { const allowed = ['unknown_trace', 'unknown_source', 'unknown_cursor', 'idempotency_conflict', 'stage_order_violation', 'used_source_required', 'unknown_confirmation', 'confirmation_id_required', 'source_ids_required']; const code = err instanceof Error && allowed.includes(err.message) ? err.message : 'invalid_memory_request'; return { content: [{ type: 'text' as const, text: JSON.stringify({ error: code }) }], isError: true }; } + }); + } +} +async function digest(s: string) { return Array.from(new Uint8Array(await crypto.subtle.digest('SHA-256', new TextEncoder().encode(s))), x => x.toString(16).padStart(2, '0')).join(''); } +export async function hashReturnedContent(content: string): Promise { return 'sha256:' + await digest(content); } +export async function identifySource(row: Record, axis: string): Promise { + const repo = String(row.repo ?? ''); const type = String(row.type ?? ''); + const path = type === 'wiki_doc' ? (row.wiki_path || row.doc_path || '') : type === 'doc' ? (row.doc_path || '') : (row.file_path || ''); + const event = type === 'pr_review' ? row.review_id : ['issue_comment', 'pr_review_comment'].includes(type) ? row.comment_id : null; + if (!repo || !type || !row.updated_at || (event !== null && !event)) throw new Error('Canonical source identity unavailable'); + const identity = ['doc', 'wiki_doc'].includes(type) ? path : type === 'diff' ? [row.commit_sha, path] : type === 'release' ? row.tag_name : event ?? row.number; + if (!identity || (Array.isArray(identity) && identity.some(x => !x))) throw new Error('Canonical source identity unavailable'); + const live = row.content_source === 'github_live'; + if (live && (!['doc', 'wiki_doc'].includes(type) || !/^sha256:[0-9a-f]{64}$/.test(row.content_version ?? ''))) throw new Error('Live source version unavailable'); + const canonical = { repo, type, identity, version: live ? row.content_version : row.updated_at, content_source: live ? 'github_live' : 'index' }; + const provenance = { ...canonical, ...(live ? { index_updated_at: row.updated_at } : {}) }; + return { source_id: 's:' + await digest(JSON.stringify(canonical)), provenance, axes: [axis], path: row.graph_path ?? [] }; +} +/** Commit a successful result before emitting its trace. No body, title, handle, or auth props is stored. */ +export async function rememberResult(env: Env, payload: any, request: unknown) { + const incomplete = payload.memory_incomplete; delete payload.memory_incomplete; + if (incomplete) { return { ...payload, memory_unavailable: true, feedback_available: false }; } + try { + const entries: { row: Record; source: SavedSource }[] = []; + for (const axis of ['results', 'graph_results']) for (const row of payload[axis] ?? []) { + entries.push({ row, source: await identifySource(row, axis === 'results' ? 'keyword' : 'graph') }); + for (const other of row.same_entity?.others ?? []) entries.push({ row: other, source: await identifySource(other, 'keyword') }); + } + const unique = new Map(); + for (const { source } of entries) { const old = unique.get(source.source_id); if (old) { old.axes = [...new Set([...old.axes, ...source.axes])]; if (source.path.length) old.path = source.path; } else unique.set(source.source_id, source); } + const { results, graph_results, ...settings } = payload; + // Settings only contain retrieval controls/outcomes, not document text. + const saved = await memoryCall(env, 'save', { request, settings, sources: [...unique.values()] }); + const snapshots = new Map(saved.sources.map((s: any) => [s.source_id, s])); + for (const { row, source } of entries) { row.source_id = source.source_id; row.provenance = source.provenance; row.activation = (snapshots.get(source.source_id) as any).activation; } + return { ...payload, trace_id: saved.trace_id, timestamp: saved.timestamp, memory_policy: saved.policy, feedback_available: true }; + } catch { return { ...payload, memory_unavailable: true, feedback_available: false }; } +} +export { MEMORY_INSTRUCTIONS }; diff --git a/src/memory-contract.test.ts b/src/memory-contract.test.ts new file mode 100644 index 0000000..77a344e --- /dev/null +++ b/src/memory-contract.test.ts @@ -0,0 +1,47 @@ +import { describe, it, expect } from 'vitest'; +import { createMcpHandler } from 'agents/mcp/server'; +import { createRemoteClient } from '../mcp-server/server/remote-client.js'; +import { createRagMcpServer } from './mcp.js'; +import type { Env } from './types.js'; +import { z } from 'zod'; +import { TOOLS } from '../mcp-server/server/tools.js'; +import { memorySchemas, memoryResultSchema, memoryDescriptions } from './memory-contract.js'; +// Bridge artifact is tested as shipped, including nested bounds and output schemas. +import { searchInputSchema, searchOutputSchema } from './search-contract.js'; +import searchMirror from '../mcp-server/server/search-schema.json'; +import mirror from '../mcp-server/server/memory-tools.json'; +describe('all memory tools Worker / bridge contract', () => { + it('mirrors exact nested input/output schemas and descriptions', () => { + for (const [name, schema] of Object.entries(memorySchemas)) { + const tool = mirror.tools.find((t: any) => t.name === name)!; + expect(tool.inputSchema).toEqual(z.toJSONSchema(schema, { io: 'input' })); + expect(tool.outputSchema).toEqual(z.toJSONSchema(memoryResultSchema)); + expect(tool.description).toBe(memoryDescriptions[name as keyof typeof memoryDescriptions]); + } + expect(mirror.tools).toHaveLength(3); + expect(searchMirror.inputSchema).toEqual(z.toJSONSchema(searchInputSchema, { io: 'input' })); + expect(searchMirror.outputSchema).toEqual(z.toJSONSchema(searchOutputSchema)); + }); + it('serves all schemas, instructions and truthful annotations over the real protocol', async () => { + const endpoint = 'https://synthetic.example/mcp'; + const handler = createMcpHandler(() => createRagMcpServer({} as Env), { route: '/mcp', legacy: 'reject' }); + const remote = createRemoteClient({ workerUrl: endpoint.replace('/mcp',''), clientVersion: 'test', fetch: (async (input: any, init: any) => { + const r = input instanceof Request ? input : new Request(input, init); const h = new Headers(r.headers); h.set('host', new URL(r.url).host); + return handler(new Request(r, { headers: h }), {} as Env, { props: { githubUserId: 9001 } } as any); + }) as typeof fetch }); + const client = await remote.getClient(); const list = (await client.listTools()).tools; + for (const expected of mirror.tools) { + const tool = list.find((t: any) => t.name === expected.name)!; + const inputSchema = expected.inputSchema; + const outputSchema = expected.outputSchema; + expect(tool.inputSchema).toEqual(inputSchema); expect(tool.outputSchema).toEqual(outputSchema); expect(tool.annotations).toEqual(expected.annotations); + } + expect(list.map((t: any) => t.name).sort()).toEqual(TOOLS.map(t => t.name).sort()); + for (const bridge of TOOLS) { const worker = list.find((t: any) => t.name === bridge.name)!; expect(worker.inputSchema).toEqual(bridge.inputSchema); expect(worker.outputSchema).toEqual(bridge.outputSchema); expect(worker.annotations).toEqual(bridge.annotations); } + const search = list.find((t: any) => t.name === 'search')!; + expect(search.inputSchema).toEqual(searchMirror.inputSchema); + expect(search.outputSchema).toEqual(searchMirror.outputSchema); + expect(search.annotations).toEqual({ readOnlyHint: false, destructiveHint: false, openWorldHint: true }); + await remote.reset(); + }); +}); diff --git a/src/memory-contract.ts b/src/memory-contract.ts new file mode 100644 index 0000000..e3ef179 --- /dev/null +++ b/src/memory-contract.ts @@ -0,0 +1,22 @@ +import { z } from 'zod'; + +export const MEMORY_INSTRUCTIONS = 'Successful search/scan/fetch writes a private trace. Retrieval is not usage or confirmation. Call record_source_use with selected (chosen for investigation), then validated (exact source checked and judged usable), then used (actually used in an answer or decision). record_outcome confirmed reinforces only saved graph paths of used sources. corrected/rolled_back names the confirmation receipt to reverse. Use a new idempotency_key for each operation; retry the same payload with the same key. memory_history lists traces or reads one trace. use_memory only orders graph candidates within the same hop; keyword scores/ranks never change. memory_unavailable means no usable trace was saved. Live inline doc/wiki bodies use returned-content SHA-256 versions and github_live provenance; stored-content fetch refers to the indexed snapshot.'; +const handle = z.string().min(1).max(128); +const reason = z.string().min(1).max(1000); +export const memorySchemas = { + memory_history: z.strictObject({ trace_id: handle.optional(), limit: z.number().int().min(1).max(50).default(10), cursor: handle.optional() }), + record_source_use: z.strictObject({ trace_id: handle, idempotency_key: handle, uses: z.array(z.strictObject({ source_id: handle, stage: z.enum(['selected', 'validated', 'used']) })).min(1).max(100) }), + record_outcome: z.strictObject({ trace_id: handle, idempotency_key: handle, outcome: z.enum(['confirmed', 'corrected', 'rolled_back']), source_ids: z.array(handle).min(1).max(100).optional(), confirmation_id: handle.optional(), reason }), +}; +export const memoryDescriptions = { + memory_history: 'Read your private, bounded trace history. trace_id returns source usage, current activation, saved paths, and outcome/reversal audit. Otherwise limit/cursor pages trace summaries.', + record_source_use: 'Atomically record selected -> validated -> used for sources in your trace. selected = chosen for investigation; validated = exact source checked and usable; used = actually used in an answer/decision. Search alone is never used. Same key/payload returns the saved receipt; conflicting payload is rejected. Repeating a completed stage adds no activation.', + record_outcome: 'confirmed requires source_ids already used in this trace and strengthens only saved actual graph paths. One trace contributes to each edge once. corrected/rolled_back requires confirmation_id and reverses only that receipt contribution, preserving other traces. reason is saved for audit. Lexical sources have no graph credit. Idempotent by key/payload.', +}; +const metadata = z.record(z.string(), z.unknown()); +const edge = z.object({ repo: z.string(), src: z.string(), dst: z.string(), kind: z.string() }); +const savedSource = z.object({ source_id: handle, provenance: metadata, axes: z.array(z.string()), path: z.array(edge).max(2) }); +const receipt = z.object({ receipt_id: handle, trace_id: handle, timestamp: z.string(), outcome: z.enum(['confirmed', 'corrected', 'rolled_back']).optional(), reason: z.string().optional(), confirmation_id: handle.nullable().optional(), uses: z.array(z.object({ source_id: handle, stage: z.enum(['selected', 'validated', 'used']) })).optional(), saved_sources: z.array(savedSource), changes: z.array(metadata), non_applied_reason: z.string().nullable(), active_credits: z.array(metadata).optional() }); +const summary = z.object({ traces: z.array(z.object({ trace_id: handle, timestamp: z.string(), request: metadata, settings: metadata })).max(50), next_cursor: handle.nullable() }); +const detail = z.object({ trace_id: handle, timestamp: z.string(), request: metadata, settings: metadata, policy: metadata, sources: z.array(savedSource.extend({ stage: z.enum(['retrieved', 'selected', 'validated', 'used']), activation: z.object({ retrieved: z.number().nonnegative(), usage: z.number().nonnegative(), total: z.number().nonnegative(), updated_at: z.string() }) })).max(300), audit: z.array(receipt).max(50), next_cursor: handle.nullable() }); +export const memoryResultSchema = z.object({ data: z.union([summary, detail, receipt]) }); diff --git a/src/memory-e2e.workers.test.ts b/src/memory-e2e.workers.test.ts new file mode 100644 index 0000000..b0739f2 --- /dev/null +++ b/src/memory-e2e.workers.test.ts @@ -0,0 +1,205 @@ +import { describe, it, expect, beforeAll, vi } from 'vitest'; +import { env, applyD1Migrations } from 'cloudflare:test'; +import { createMcpHandler } from 'agents/mcp/server'; +import { createRagMcpServer } from './mcp.js'; +import { createRemoteClient } from '../mcp-server/server/remote-client.js'; +import { upsertFtsRow, queryFts } from './fts.js'; +import { upsertEdges, queryNeighbors, deleteEdgesForVector } from './graph.js'; +import { backfillSourceIdentities } from './backfill-source-identities.js'; +import { buildFetchItem } from './fetch.js'; +import { identifySource } from './memory-api.js'; +import { computeBodyHash } from './pipeline/hash.js'; +import { issueCommentVectorId, prReviewVectorId, prReviewCommentVectorId } from './pipeline/vector-id.js'; +import { ingestIssueComment, ingestPRReview, ingestPRReviewComment } from './pipeline/embed-comment.js'; +import type { Env } from './types.js'; +beforeAll(() => applyD1Migrations(env.DB_FTS, env.TEST_MIGRATIONS)); +const timestamp = '2026-01-01T00:00:00Z'; +const seed = async (repo: string, id: string, slug: string, content: string, type: 'wiki_doc' | 'doc' = 'wiki_doc') => upsertFtsRow(env.DB_FTS, { vectorId: id, repo, type, state: 'active', labels: '', milestone: '', assignees: '', updatedAt: timestamp, docPath: slug, content }); +async function client(principal = 9101, failMemory = false, failScan = false) { + const bindings = { DB_FTS: env.DB_FTS, ISSUE_STORE: failMemory ? { idFromName: () => 'global', get: () => ({ fetch: async () => new Response('unavailable', { status: 503 }) }) } : failScan ? { idFromName: (name: string) => env.ISSUE_STORE.idFromName(name), get: (id: DurableObjectId) => { const real = env.ISSUE_STORE.get(id); return { fetch: (r: Request) => new URL(r.url).pathname === '/recent-reviews' ? Promise.resolve(new Response('synthetic unavailable', {status:503})) : real.fetch(r) }; } } : env.ISSUE_STORE } as unknown as Env; + const handler = createMcpHandler(() => createRagMcpServer(bindings), { route: '/mcp', legacy: 'reject' }); + const remote = createRemoteClient({ workerUrl: 'https://synthetic.example', clientVersion: 'test', fetch: (async (input: any, init: any) => { + const r = input instanceof Request ? input : new Request(input, init); const headers = new Headers(r.headers); headers.set('host', new URL(r.url).host); + return handler(new Request(r, { headers }), bindings, { props: { githubUserId: principal, accessToken: 'synthetic-test-only' } } as any); + }) as typeof fetch }); + const call = async (name: string, args: unknown) => { const result = await remote.callTool(name, args) as any; return { error: result.isError ?? false, payload: result.isError ? { error: result.content[0].text.startsWith('{') ? JSON.parse(result.content[0].text).error : result.content[0].text } : JSON.parse(result.content[0].text), structured: result.structuredContent }; }; + return { call, reset: () => remote.reset() }; +} +const uses = (source_id: string) => ['selected', 'validated', 'used'].map(stage => ({ source_id, stage })); +describe('synthetic in-process MCP lifecycle with real DO/D1', () => { + it('search -> explicit used -> confirmed two-hop path -> corrected; source versions survive vector migration', async () => { + const repo = 'synthetic/e2e-path'; + await seed(repo, 'e2e-a', 'a', 'syntheticneedle'); await seed(repo, 'e2e-b', 'b', 'middle page'); await seed(repo, 'e2e-c', 'c', 'terminal page'); + await upsertEdges(env.DB_FTS, 'e2e-a', repo, 'a', [{ dstVectorId: 'e2e-b', dstSlug: 'b', edgeKind: 'mention' }]); + await upsertEdges(env.DB_FTS, 'e2e-b', repo, 'b', [{ dstVectorId: 'e2e-c', dstSlug: 'c', edgeKind: 'mention' }]); + const a = await client(); const b = await client(); const other = await client(9102); + const found = (await a.call('search', { query: 'syntheticneedle', repo, fusion: 'sparse_only', rerank: false, graph_expand: true, graph_hops: 2 })).payload; + expect(found.feedback_available).toBe(true); expect(found.timestamp).toMatch(/Z$/); expect(found.graph_results.map((r: any) => r.graph_hop)).toEqual([1,2]); + const terminal = found.graph_results.find((r: any) => r.wiki_path === 'c'); expect(terminal.graph_path.map((e: any) => [e.src,e.dst])).toEqual([['a','b'],['b','c']]); + expect((await a.call('record_outcome', { trace_id: found.trace_id, idempotency_key: 'premature', outcome: 'confirmed', source_ids: [terminal.source_id], reason: 'synthetic' })).payload.error).toBe('used_source_required'); + const usageArgs = { trace_id: found.trace_id, idempotency_key: 'e2e-used', uses: uses(terminal.source_id) }; + const usage = await a.call('record_source_use', usageArgs); expect(usage.error).toBe(false); expect((await b.call('record_source_use', usageArgs)).payload).toEqual(usage.payload); + const confirmed = (await a.call('record_outcome', { trace_id: found.trace_id, idempotency_key: 'e2e-confirmed', outcome: 'confirmed', source_ids: [terminal.source_id], reason: 'Synthetic exact source checked' })).payload.data; + expect(confirmed.changes).toHaveLength(2); + const learned = (await b.call('search', { query: 'syntheticneedle', repo, fusion: 'sparse_only', rerank: false, graph_expand: true, graph_hops: 2, use_memory: true })).payload; + expect(learned.results.map((r: any) => [r.source_id,r.score,r.dense_rank,r.sparse_rank])).toEqual(found.results.map((r: any) => [r.source_id,r.score,r.dense_rank,r.sparse_rank])); + expect(learned.graph_results.find((r: any) => r.wiki_path === 'c').learned_strength).toBeGreaterThan(0.49); + const corrected = (await b.call('record_outcome', { trace_id: found.trace_id, idempotency_key: 'e2e-corrected', outcome: 'corrected', confirmation_id: confirmed.receipt_id, reason: 'Synthetic decision corrected' })).payload.data; + expect(corrected.changes).toHaveLength(2); + const history = (await b.call('memory_history', { trace_id: found.trace_id })).payload.data; + expect(history.sources.find((s: any) => s.source_id === terminal.source_id).stage).toBe('used'); + expect(history.audit.find((r: any) => r.receipt_id === confirmed.receipt_id).active_credits.every((c: any) => c.active === 0)).toBe(true); + expect((await other.call('memory_history', { trace_id: found.trace_id })).payload.error).toBe('unknown_trace'); + expect((await other.call('record_outcome', { trace_id: found.trace_id, idempotency_key: 'cross-user', outcome: 'rolled_back', confirmation_id: confirmed.receipt_id, reason: 'synthetic' })).payload.error).toBe('unknown_trace'); + const fetch = (await a.call('search', { vector_ids: ['e2e-c'] })).payload; expect(fetch.mode).toBe('fetch'); expect(fetch.results[0].source_id).toBe(terminal.source_id); + const empty = (await a.call('search', { query: 'absentuniquetoken', repo, fusion: 'sparse_only', rerank: false })).payload; expect(empty.count).toBe(0); expect(empty.trace_id).toBeTruthy(); + const scan = (await a.call('search', { repo, since: '2025-01-01', until: '2027-01-01' })).payload; expect(scan.mode).toBe('scan'); expect(scan.trace_id).toBeTruthy(); + await seed(repo, 'e2e-c-migrated', 'c', 'terminal page'); await deleteEdgesForVector(env.DB_FTS, 'e2e-c'); + await upsertEdges(env.DB_FTS, 'e2e-b', repo, 'b', [{ dstVectorId: 'e2e-c-migrated', dstSlug: 'c', edgeKind: 'mention' }]); + const migrated = (await a.call('search', { vector_ids: ['e2e-c-migrated'] })).payload; expect(migrated.results[0].source_id).toBe(terminal.source_id); + expect((await b.call('memory_history', { trace_id: found.trace_id })).payload.data.sources.find((s: any) => s.source_id === terminal.source_id).path).toEqual(terminal.graph_path); + await a.reset(); await b.reset(); await other.reset(); + }); + it('memory failure emits no fabricated trace and input errors do not save a successful trace', async () => { + const c = await client(9103, true); + const result = (await c.call('search', { vector_ids: ['missing-synthetic'] })).payload; + expect(result.memory_unavailable).toBe(true); expect(result.feedback_available).toBe(false); expect(result.trace_id).toBeUndefined(); + await c.reset(); + const normal = await client(9104); + const before = (await normal.call('memory_history', {})).payload.data.traces.length; + expect((await normal.call('search', { query: 'synthetic', path_prefix: '../bad/', type: 'doc' })).error).toBe(true); + expect((await normal.call('memory_history', {})).payload.data.traces.length).toBe(before); + await normal.reset(); + }); + it('learned same-hop candidate outside old output cap enters bounded pool; initial/default axes match', async () => { + const repo = 'synthetic/e2e-pool'; await seed(repo, 'pool-seed', 'seed', 'poolneedle'); + const edges = []; + for (let i = 0; i < 40; i++) { const id = 'pool-' + String(i).padStart(3,'0'); await seed(repo, id, id, 'relatedpage ' + (i === 39 ? 'lastneedle' : 'othertext')); edges.push({ dstVectorId: id, dstSlug: id, edgeKind: 'mention' }); } + await upsertEdges(env.DB_FTS, 'pool-seed', repo, 'seed', edges); + const c = await client(9105); const searchArgs = { query: 'poolneedle', repo, fusion: 'sparse_only', rerank: false, graph_expand: true, top_k: 1 }; + const standard = (await c.call('search', searchArgs)).payload; + const initial = (await c.call('search', { ...searchArgs, use_memory: true })).payload; + expect(initial.graph_results.map((s: any) => s.source_id)).toEqual(standard.graph_results.map((s: any) => s.source_id)); + // Find the same mention in reverse traversal, then credit the real seed->last edge. + const reverse = (await c.call('search', { ...searchArgs, query: 'lastneedle' })).payload; + const seedResult = reverse.graph_results.find((r: any) => r.wiki_path === 'seed'); + await c.call('record_source_use', { trace_id: reverse.trace_id, idempotency_key: 'pool-used', uses: uses(seedResult.source_id) }); + await c.call('record_outcome', { trace_id: reverse.trace_id, idempotency_key: 'pool-confirm', outcome: 'confirmed', source_ids: [seedResult.source_id], reason: 'Synthetic mention verified' }); + const learned = (await c.call('search', { ...searchArgs, use_memory: true })).payload; + expect(learned.graph_results[0].wiki_path).toBe('pool-039'); expect(learned.graph_results).toHaveLength(2); + expect(learned.graph_results.every((r: any) => r.score === undefined)).toBe(true); + expect(learned.results.map((r: any) => [r.vector_id,r.score])).toEqual(standard.results.map((r: any) => [r.vector_id,r.score])); + const unchanged = (await c.call('search', searchArgs)).payload; expect(unchanged.graph_results.map((r: any) => r.wiki_path)).toEqual(standard.graph_results.map((r: any) => r.wiki_path)); + // Reindex same topology never resets or cross-contaminates private learned state. + await upsertEdges(env.DB_FTS, 'pool-seed', repo, 'seed', edges); + expect((await c.call('search', { ...searchArgs, use_memory: true })).payload.graph_results[0].wiki_path).toBe('pool-039'); + const stranger = await client(9106); expect((await stranger.call('search', { ...searchArgs, use_memory: true })).payload.graph_results.map((s: any) => s.source_id)).toEqual(standard.graph_results.map((s: any) => s.source_id)); + await c.reset(); await stranger.reset(); + }); + it('event IDs repair unchanged legacy rows without embeddings, and versions/events never alias', async () => { + const repo = 'synthetic/e2e-identity'; const comment = { id: 90001, body: 'Synthetic canonical comment body long enough for indexing.', user: { login: 'synthetic' }, created_at: timestamp, updated_at: timestamp, path: 'a.ts', line: 1, commit_id: 'syntheticsha' }; + const review = { ...comment, id: 90002, state: 'APPROVED', submitted_at: timestamp }; + const variants = [ + { id: await issueCommentVectorId(repo, comment.id), type: 'issue_comment' as const, ingest: () => ingestIssueComment(bindings, stub, repo, 1, comment), event: comment.id }, + { id: await prReviewVectorId(repo, review.id), type: 'pr_review' as const, ingest: () => ingestPRReview(bindings, stub, repo, 1, review), event: review.id }, + { id: await prReviewCommentVectorId(repo, comment.id), type: 'pr_review_comment' as const, ingest: () => ingestPRReviewComment(bindings, stub, repo, 1, comment), event: comment.id }, + ]; + const hash = await computeBodyHash('synthetic', comment.body); + const stub = { fetch: async (r: Request) => Response.json({ bodyHash: r.url.includes('/review?') ? await computeBodyHash('synthetic\n\nAPPROVED', comment.body) : r.url.includes('/review-comment?') ? await computeBodyHash('synthetic\na.ts:1', comment.body) : hash }) } as unknown as DurableObjectStub; + const bindings = { DB_FTS: env.DB_FTS } as Env; + for (const v of variants) { + await upsertFtsRow(env.DB_FTS, { vectorId: v.id, repo, type: v.type, state: 'active', labels: '', milestone: '', assignees: '', updatedAt: timestamp, number: 1, content: comment.body }); + const result = await v.ingest(); expect(result.skippedUnchanged).toBe(true); + const hit = (await queryFts(env.DB_FTS, 'canonical', 50, { repo, type: v.type }))[0]; expect(v.type === 'pr_review' ? hit.reviewId : hit.commentId).toBe(v.event); + } + const first = await identifySource({ repo, type: 'issue_comment', number: 1, comment_id: 1, updated_at: timestamp, vector_id: 'old' }, 'keyword'); + expect((await identifySource({ repo, type: 'issue_comment', number: 1, comment_id: 1, updated_at: timestamp, vector_id: 'new' }, 'keyword')).source_id).toBe(first.source_id); + expect((await identifySource({ repo, type: 'issue_comment', number: 1, comment_id: 2, updated_at: timestamp }, 'keyword')).source_id).not.toBe(first.source_id); + expect((await identifySource({ repo, type: 'issue_comment', number: 1, comment_id: 1, updated_at: '2026-01-02T00:00:00Z' }, 'keyword')).source_id).not.toBe(first.source_id); + }); + it('one failed scan surface returns retrieval but never saves a partial success trace', async () => { + const repo = 'synthetic/partial-scan'; + const stub = env.ISSUE_STORE.get(env.ISSUE_STORE.idFromName('global')); + await stub.fetch(new Request('http://store/upsert', { method:'POST', body:JSON.stringify({ repo, number:1, type:'issue', state:'open', title:'synthetic', labels:[], milestone:'', assignees:[], bodyHash:'synthetic', createdAt:timestamp, updatedAt:timestamp }) })); + const c = await client(9107, false, true); + const result = (await c.call('search', { repo, since:'2025-01-01', until:'2027-01-01' })).payload; + expect(result.results).toHaveLength(1); expect(result.memory_unavailable).toBe(true); expect(result.feedback_available).toBe(false); expect(result.trace_id).toBeUndefined(); + expect((await c.call('memory_history', {})).payload.data.traces).toEqual([]); + await c.reset(); + }); + it('folded doc/diff rows retain complete provenance and return separate version handles', async () => { + const repo = 'synthetic/folded-provenance'; + await seed(repo, 'fold-doc', 'docs/a.md', 'foldneedle foldneedle documentation', 'doc'); + await upsertFtsRow(env.DB_FTS, { vectorId:'fold-diff', repo, type:'diff', state:'committed', labels:'', milestone:'', assignees:'', updatedAt:timestamp, filePath:'docs/a.md', commitSha:'syntheticcommit', content:'foldneedle' }); + const c = await client(9108); const result = (await c.call('search', { query:'foldneedle', repo, fusion:'sparse_only', rerank:false })).payload; + expect(result.feedback_available).toBe(true); expect(result.results).toHaveLength(1); + const primary = result.results[0]; const secondary = primary.same_entity.others[0]; + expect(primary.source_id).toBeTruthy(); expect(secondary.source_id).toBeTruthy(); expect(secondary.source_id).not.toBe(primary.source_id); + const fetched = (await c.call('search', { vector_ids:[primary.vector_id, secondary.vector_id] })).payload; + expect(fetched.results.map((r: any) => r.source_id)).toEqual([primary.source_id, secondary.source_id]); + await c.reset(); + }); + it('source handles use canonical kind identity/version across all modes; optional fields never alter them', async () => { + for (const type of ['issue', 'pull_request', 'release', 'doc', 'wiki_doc', 'diff', 'issue_comment', 'pr_review', 'pr_review_comment']) { + const row = { repo:'synthetic/all-modes', type, number:11, updated_at:timestamp, doc_path:'docs/a.md', wiki_path:'docs/a.md', file_path:'a.ts', commit_sha:'syntheticsha', tag_name:'synthetic-v1', comment_id:123, review_id:124, content:'synthetic' }; + const original = await identifySource(row, 'keyword'); + const fetchRow = buildFetchItem('arbitrary-vector', row); + expect((await identifySource(fetchRow, 'keyword')).source_id).toBe(original.source_id); + const { content, commit_sha, ...scanRow } = row; + expect((await identifySource(type === 'diff' ? row : scanRow, 'keyword')).source_id).toBe(original.source_id); + } + }); + it('bounded historical ID backfill repairs unchanged stored events atomically and idempotently', async () => { + const repo = 'synthetic/historical-identity'; const stub = env.ISSUE_STORE.get(env.ISSUE_STORE.idFromName(crypto.randomUUID())); + for (let id=1; id<=3; id++) { + await stub.fetch(new Request('http://store/upsert-comment', { method:'POST', body:JSON.stringify({repo, commentId:id, number:1, author:'synthetic', bodyHash:'unchanged', createdAt:timestamp, updatedAt:timestamp}) })); + await upsertFtsRow(env.DB_FTS, { vectorId:await issueCommentVectorId(repo,id), repo, type:'issue_comment', state:'active', labels:'', milestone:'', assignees:'', updatedAt:timestamp, number:1, content:'synthetic unchanged historical comment' }); + } + const page1 = await backfillSourceIdentities(env.DB_FTS, stub, { repo, limit:2 }); expect(page1).toEqual({ scanned:2, updated:2, next_cursor:'issue_comment:2', done:false }); + const page2 = await backfillSourceIdentities(env.DB_FTS, stub, { repo, limit:2, cursor:page1.next_cursor! }); expect(page2).toEqual({ scanned:1, updated:1, next_cursor:null, done:true }); + expect((await backfillSourceIdentities(env.DB_FTS, stub, { repo, limit:2 })).updated).toBe(0); + expect((await queryFts(env.DB_FTS, 'historical', 10, {repo})).map(r=>r.commentId).sort()).toEqual([1,2,3]); + }); + + it('live inline doc/wiki body revisions get separate source IDs while index timestamp and keyword ranks stay fixed', async () => { + const repo = 'synthetic/live-version'; + await seed(repo, 'live-doc', 'docs/live.md', 'liveneedle indexed original', 'doc'); + await seed(repo, 'live-wiki', 'live-page', 'liveneedle indexed original'); + let liveText = 'synthetic live body version one'; + vi.stubGlobal('fetch', async (input: RequestInfo | URL) => { + const url = input instanceof Request ? input.url : String(input); + if (url.startsWith('https://api.github.com/repos/synthetic/live-version/contents/')) return Response.json({ encoding:'base64', content:btoa(liveText) }); + if (url.startsWith('https://raw.githubusercontent.com/wiki/synthetic/live-version/')) return new Response(liveText); + throw new Error('Unexpected synthetic fixture request'); + }); + const c = await client(9109); + try { + const args = { query:'liveneedle', repo, fusion:'sparse_only', rerank:false }; + const indexed = (await c.call('search', args)).payload; + const first = (await c.call('search', { ...args, include_content:true })).payload; + const retry = (await c.call('search', { ...args, include_content:true })).payload; + expect(first.results).toHaveLength(2); expect(first.feedback_available).toBe(true); + expect(first.results.map((r: any) => [r.vector_id,r.score,r.sparse_rank])).toEqual(indexed.results.map((r: any) => [r.vector_id,r.score,r.sparse_rank])); + expect(retry.results.map((r: any) => r.source_id)).toEqual(first.results.map((r: any) => r.source_id)); + liveText = 'synthetic live body version two'; + const second = (await c.call('search', { ...args, include_content:true })).payload; + for (const initial of first.results) { + const changed = second.results.find((r: any) => r.vector_id === initial.vector_id); + const snapshot = indexed.results.find((r: any) => r.vector_id === initial.vector_id); + expect(changed.updated_at).toBe(initial.updated_at); expect(initial.updated_at).toBe(timestamp); + expect(changed.source_id).not.toBe(initial.source_id); expect(initial.source_id).not.toBe(snapshot.source_id); + expect(initial.content_source).toBe('github_live'); expect(initial.content_version).toMatch(/^sha256:[a-f0-9]{64}$/); + expect(initial.provenance.version).toBe(initial.content_version); expect(initial.provenance.index_updated_at).toBe(timestamp); + expect(snapshot.provenance.content_source).toBe('index'); expect(changed.provenance.content_source).toBe('github_live'); + } + const fetched = (await c.call('search', { vector_ids:['live-doc','live-wiki'] })).payload; + expect(fetched.results.map((r: any) => r.source_id)).toEqual(['live-doc','live-wiki'].map(id => indexed.results.find((r: any) => r.vector_id === id).source_id)); + const h = (await c.call('memory_history', { trace_id:first.trace_id })).payload.data; + expect(h.sources.every((s: any) => s.provenance.content_source === 'github_live')).toBe(true); + expect(JSON.stringify(h)).not.toContain('synthetic live body version one'); + await c.call('record_source_use', { trace_id:first.trace_id, idempotency_key:'live-used', uses:uses(first.results[0].source_id) }); + expect((await c.call('memory_history', { trace_id:second.trace_id })).payload.data.sources.every((s: any) => s.stage === 'retrieved')).toBe(true); + } finally { vi.unstubAllGlobals(); await c.reset(); } + }); + +}); diff --git a/src/memory.ts b/src/memory.ts new file mode 100644 index 0000000..3459fe8 --- /dev/null +++ b/src/memory.ts @@ -0,0 +1,139 @@ +/** Private SQLite memory. Every mutation runs inside the owning DO transactionSync. */ +import { memorySchemas } from './memory-contract.js'; +export const MEMORY_POLICY = { half_life_seconds: 3600, activation_cap: 10, retrieved: 0.1, selected: 0.2, validated: 0.5, used: 1, strength_cap: 5, confirmed_delta: 0.25 } as const; +export interface SavedEdge { repo: string; src: string; dst: string; kind: string } +export interface SavedSource { source_id: string; provenance: Record; axes: string[]; path: SavedEdge[] } +export function decay(value: number, updated: number, now: number): number { + return Math.max(0, value) * Math.pow(0.5, Math.max(0, now - updated) / (MEMORY_POLICY.half_life_seconds * 1000)); +} +export function edgeKey(e: SavedEdge): string { return JSON.stringify([e.repo, e.src, e.dst, e.kind]); } +const stages = ['selected', 'validated', 'used']; +function canonical(x: unknown): string { + if (Array.isArray(x)) return '[' + x.map(canonical).join(',') + ']'; + if (x && typeof x === 'object') return '{' + Object.entries(x).sort(([a], [b]) => a.localeCompare(b)).map(([k, v]) => JSON.stringify(k) + ':' + canonical(v)).join(',') + '}'; + return JSON.stringify(x); +} +export class MemoryStore { + constructor(private sql: SqlStorage, private transaction: (fn: () => T) => T, private clock = () => Date.now()) { + sql.exec(`CREATE TABLE IF NOT EXISTS memory_traces (seq INTEGER PRIMARY KEY AUTOINCREMENT, principal TEXT NOT NULL, trace_id TEXT NOT NULL UNIQUE, timestamp TEXT NOT NULL, request TEXT NOT NULL, settings TEXT NOT NULL, sources TEXT NOT NULL); + CREATE INDEX IF NOT EXISTS memory_trace_owner ON memory_traces(principal,seq); + CREATE TABLE IF NOT EXISTS memory_usage (principal TEXT NOT NULL, trace_id TEXT NOT NULL, source_id TEXT NOT NULL, stage INTEGER NOT NULL, PRIMARY KEY(principal,trace_id,source_id)); + CREATE TABLE IF NOT EXISTS memory_activation (principal TEXT NOT NULL, source_id TEXT NOT NULL, retrieved REAL NOT NULL, usage REAL NOT NULL, updated REAL NOT NULL, PRIMARY KEY(principal,source_id)); + CREATE TABLE IF NOT EXISTS memory_receipts (seq INTEGER PRIMARY KEY AUTOINCREMENT, principal TEXT NOT NULL, trace_id TEXT NOT NULL, key TEXT NOT NULL, payload TEXT NOT NULL, receipt_id TEXT NOT NULL UNIQUE, data TEXT NOT NULL, UNIQUE(principal,key)); + CREATE INDEX IF NOT EXISTS memory_receipt_trace ON memory_receipts(principal,trace_id,seq); + CREATE TABLE IF NOT EXISTS memory_edges (principal TEXT NOT NULL, edge TEXT NOT NULL, strength REAL NOT NULL, updated REAL NOT NULL, PRIMARY KEY(principal,edge)); + CREATE TABLE IF NOT EXISTS memory_credits (principal TEXT NOT NULL, trace_id TEXT NOT NULL, receipt_id TEXT NOT NULL, edge TEXT NOT NULL, delta REAL NOT NULL, timestamp REAL NOT NULL, active INTEGER NOT NULL, PRIMARY KEY(principal,trace_id,edge)); + CREATE INDEX IF NOT EXISTS memory_credit_receipt ON memory_credits(principal,receipt_id);`); + } + private rows(query: string, ...args: (string | number)[]) { return this.sql.exec(query, ...args).toArray(); } + private trace(p: string, id: string) { + const t = this.rows('SELECT * FROM memory_traces WHERE principal=? AND trace_id=?', p, id)[0]; + if (!t) throw new Error('Unknown trace'); + return { ...t, trace_id: t.trace_id, timestamp: t.timestamp, request: t.request, settings: t.settings, sources: JSON.parse(String(t.sources)) as SavedSource[] }; + } + private activation(p: string, id: string, now: number) { + const a = this.rows('SELECT * FROM memory_activation WHERE principal=? AND source_id=?', p, id)[0]; + return { retrieved: a ? decay(Number(a.retrieved), Number(a.updated), now) : 0, usage: a ? decay(Number(a.usage), Number(a.updated), now) : 0, updated: a ? Number(a.updated) : now }; + } + private addActivation(p: string, id: string, channel: 'retrieved' | 'usage', amount: number, now: number) { + const a = this.activation(p, id, now); + // Each channel has its own cap, making retrieval and explicit usage inspectable. + a[channel] = Math.min(MEMORY_POLICY.activation_cap, a[channel] + amount); + this.sql.exec('INSERT OR REPLACE INTO memory_activation VALUES (?,?,?,?,?)', p, id, a.retrieved, a.usage, Math.max(a.updated, now)); + } + private current(p: string, s: SavedSource, now: number) { + const a = this.activation(p, s.source_id, now); + return { ...s, activation: { retrieved: a.retrieved, usage: a.usage, total: a.retrieved + a.usage, updated_at: new Date(a.updated).toISOString() } }; + } + save(p: string, input: { request: unknown; settings: unknown; sources: SavedSource[] }) { + if (!Array.isArray(input.sources) || input.sources.length > 300 || JSON.stringify(input).length > 300000) throw new Error('Trace bound exceeded'); + return this.transaction(() => { + const now = this.clock(); const trace_id = crypto.randomUUID(); + const unique = [...new Map(input.sources.map(s => [s.source_id, s])).values()]; + this.sql.exec('INSERT INTO memory_traces(principal,trace_id,timestamp,request,settings,sources) VALUES(?,?,?,?,?,?)', p, trace_id, new Date(now).toISOString(), JSON.stringify(input.request), JSON.stringify(input.settings), JSON.stringify(unique)); + for (const s of unique) this.addActivation(p, s.source_id, 'retrieved', MEMORY_POLICY.retrieved, now); + return { trace_id, timestamp: new Date(now).toISOString(), policy: MEMORY_POLICY, sources: unique.map(s => this.current(p, s, now)) }; + }); + } + history(p: string, raw: unknown) { + const args = memorySchemas.memory_history.parse(raw); const now = this.clock(); + if (args.trace_id) { + const t = this.trace(p, args.trace_id); + const usage = this.rows('SELECT source_id,stage FROM memory_usage WHERE principal=? AND trace_id=?', p, args.trace_id); + // Audit is separately paged; never scan the entire receipt history. + const cursor = args.cursor ? this.receiptCursor(p, args.trace_id, args.cursor) : 0; + const audit = this.rows('SELECT * FROM memory_receipts WHERE principal=? AND trace_id=? AND seq>? ORDER BY seq LIMIT ?', p, args.trace_id, cursor, args.limit + 1); + return { trace_id: t.trace_id, timestamp: t.timestamp, request: JSON.parse(String(t.request)), settings: JSON.parse(String(t.settings)), policy: MEMORY_POLICY, + sources: t.sources.map(s => ({ ...this.current(p, s, now), stage: stages[Number(usage.find(u => u.source_id === s.source_id)?.stage ?? -1)] ?? 'retrieved' })), + audit: audit.slice(0, args.limit).map(r => ({ ...JSON.parse(String(r.data)), active_credits: this.rows('SELECT edge,delta,timestamp,active FROM memory_credits WHERE principal=? AND receipt_id=?', p, String(r.receipt_id)) })), + next_cursor: audit.length > args.limit ? audit[args.limit - 1].receipt_id : null }; + } + let before = Number.MAX_SAFE_INTEGER; + if (args.cursor) { const c = this.rows('SELECT seq FROM memory_traces WHERE principal=? AND trace_id=?', p, args.cursor)[0]; if (!c) throw new Error('Unknown cursor'); before = Number(c.seq); } + const list = this.rows('SELECT trace_id,timestamp,request,settings FROM memory_traces WHERE principal=? AND seq ({ ...t, request: JSON.parse(String(t.request)), settings: JSON.parse(String(t.settings)) })), next_cursor: list.length > args.limit ? list[args.limit - 1].trace_id : null }; + } + private receiptCursor(p: string, trace: string, id: string) { const r = this.rows('SELECT seq FROM memory_receipts WHERE principal=? AND trace_id=? AND receipt_id=?', p, trace, id)[0]; if (!r) throw new Error('Unknown cursor'); return Number(r.seq); } + strengths(p: string, paths: SavedEdge[][]) { + if (paths.length > 200 || paths.some(path => path.length > 2)) throw new Error('Path bound exceeded'); + const now = this.clock(); + return paths.map(path => path.reduce((sum, e) => { const r = this.rows('SELECT strength,updated FROM memory_edges WHERE principal=? AND edge=?', p, edgeKey(e))[0]; return sum + (r ? decay(Number(r.strength), Number(r.updated), now) : 0); }, 0)); + } + mutate(p: string, tool: 'record_source_use' | 'record_outcome', raw: unknown) { + const args = memorySchemas[tool].parse(raw); const payload = canonical({ tool, args }); + return this.transaction(() => { + const old = this.rows('SELECT payload,data FROM memory_receipts WHERE principal=? AND key=?', p, args.idempotency_key)[0]; + if (old) { if (old.payload !== payload) throw new Error('Idempotency conflict'); return JSON.parse(String(old.data)); } + const t = this.trace(p, args.trace_id); const now = this.clock(); const receipt_id = crypto.randomUUID(); + const changes: unknown[] = []; + if (tool === 'record_source_use') { + const a = memorySchemas.record_source_use.parse(raw); + for (const u of a.uses) { + const s = t.sources.find(s => s.source_id === u.source_id); if (!s) throw new Error('Unknown source in trace'); + const prev = this.rows('SELECT stage FROM memory_usage WHERE principal=? AND trace_id=? AND source_id=?', p, a.trace_id, u.source_id)[0]; + const stage = stages.indexOf(u.stage); const current = prev ? Number(prev.stage) : -1; + if (stage <= current) { changes.push({ ...u, applied: false }); continue; } + if (stage !== current + 1) throw new Error('Usage stage order violation'); + this.sql.exec('INSERT OR REPLACE INTO memory_usage VALUES(?,?,?,?)', p, a.trace_id, u.source_id, stage); + this.addActivation(p, u.source_id, 'usage', MEMORY_POLICY[u.stage], now); changes.push({ ...u, applied: true }); + } + } else { + const a = memorySchemas.record_outcome.parse(raw); + if (a.outcome === 'confirmed') { + if (!a.source_ids || a.confirmation_id) throw new Error('confirmed requires source_ids only'); + const edges = new Map(); + for (const id of a.source_ids) { + const s = t.sources.find(s => s.source_id === id); if (!s) throw new Error('Unknown source in trace'); + const u = this.rows('SELECT stage FROM memory_usage WHERE principal=? AND trace_id=? AND source_id=?', p, a.trace_id, id)[0]; + if (Number(u?.stage ?? -1) !== 2) throw new Error('Confirmation requires used source'); + for (const e of s.path) edges.set(edgeKey(e), e); + } + for (const [key, e] of edges) { + if (this.rows('SELECT 1 FROM memory_credits WHERE principal=? AND trace_id=? AND edge=?', p, a.trace_id, key).length) { changes.push({ edge: e, applied: false, reason: 'trace_already_credited' }); continue; } + const r = this.rows('SELECT * FROM memory_edges WHERE principal=? AND edge=?', p, key)[0]; const at = Math.max(now, Number(r?.updated ?? now)); + const strength = r ? decay(Number(r.strength), Number(r.updated), at) : 0; const delta = Math.min(MEMORY_POLICY.confirmed_delta, Math.max(0, MEMORY_POLICY.strength_cap - strength)); + this.sql.exec('INSERT OR REPLACE INTO memory_edges VALUES(?,?,?,?)', p, key, strength + delta, at); + this.sql.exec('INSERT INTO memory_credits VALUES(?,?,?,?,?,?,1)', p, a.trace_id, receipt_id, key, delta, at); + changes.push({ edge: e, delta, reason: delta > 0 ? 'confirmed_path' : 'strength_cap_reached', timestamp: new Date(at).toISOString(), applied: delta > 0 }); + } + } else { + if (!a.confirmation_id || a.source_ids) throw new Error('Reversal requires confirmation_id only'); + const r = this.rows('SELECT data FROM memory_receipts WHERE principal=? AND trace_id=? AND receipt_id=?', p, a.trace_id, a.confirmation_id)[0]; + if (!r || JSON.parse(String(r.data)).outcome !== 'confirmed') throw new Error('Unknown confirmation for trace'); + const credits = this.rows('SELECT * FROM memory_credits WHERE principal=? AND receipt_id=?', p, a.confirmation_id); + for (const c of credits) { + if (!c.active) continue; + const edge = this.rows('SELECT * FROM memory_edges WHERE principal=? AND edge=?', p, String(c.edge))[0]; const at = Math.max(now, Number(edge.updated)); + const removed = decay(Number(c.delta), Number(c.timestamp), at); + this.sql.exec('UPDATE memory_edges SET strength=?,updated=? WHERE principal=? AND edge=?', Math.max(0, decay(Number(edge.strength), Number(edge.updated), at) - removed), at, p, String(c.edge)); + this.sql.exec('UPDATE memory_credits SET active=0 WHERE principal=? AND trace_id=? AND edge=?', p, a.trace_id, String(c.edge)); + changes.push({ edge: JSON.parse(String(c.edge)), reversed_delta: removed, confirmation_id: a.confirmation_id }); + } + } + } + const data = { non_applied_reason: changes.length === 0 ? ('outcome' in args && args.outcome === 'confirmed' ? 'no_graph_path' : 'no_active_credits') : null, receipt_id, trace_id: args.trace_id, timestamp: new Date(now).toISOString(), ...('outcome' in args ? { outcome: args.outcome, reason: args.reason, confirmation_id: args.confirmation_id ?? null } : { uses: memorySchemas.record_source_use.parse(raw).uses }), saved_sources: t.sources.filter(s => 'source_ids' in args && args.source_ids?.includes(s.source_id)), changes }; + this.sql.exec('INSERT INTO memory_receipts(principal,trace_id,key,payload,receipt_id,data) VALUES(?,?,?,?,?,?)', p, args.trace_id, args.idempotency_key, payload, receipt_id, JSON.stringify(data)); + return data; + }); + } +} diff --git a/src/memory.workers.test.ts b/src/memory.workers.test.ts new file mode 100644 index 0000000..b57b3fe --- /dev/null +++ b/src/memory.workers.test.ts @@ -0,0 +1,84 @@ +import { describe, it, expect } from 'vitest'; +import { env, runInDurableObject } from 'cloudflare:test'; +import { MemoryStore, decay, edgeKey, type SavedSource, type SavedEdge } from './memory.js'; +const path: SavedEdge[] = [{ repo: 'synthetic/repo', src: 'a', dst: 'b', kind: 'mention' }, { repo: 'synthetic/repo', src: 'b', dst: 'c', kind: 'mention' }]; +const source = (id = 's:synthetic'): SavedSource => ({ source_id: id, provenance: { repo: 'synthetic/repo', type: 'wiki_doc', identity: 'c', version: '2026-01-01T00:00:00Z' }, axes: ['graph'], path }); +async function fixture(fn: (m: MemoryStore, advance: (ms: number) => void, sql: SqlStorage) => void) { + const stub = env.ISSUE_STORE.get(env.ISSUE_STORE.idFromName(crypto.randomUUID())); + await runInDurableObject(stub, (_instance, state) => { let now = Date.parse('2026-01-01T00:00:00Z'); const m = new MemoryStore(state.storage.sql, f => state.storage.transactionSync(f), () => now); fn(m, ms => { now += ms; }, state.storage.sql); }); +} +function save(m: MemoryStore, p = 'github:9001', sources = [source()], mode = 'search') { return m.save(p, { request: { query: 'synthetic query', mode }, settings: { mode, fusion: 'rrf' }, sources }); } +function use(m: MemoryStore, trace: string, ids = ['s:synthetic']) { + return m.mutate('github:9001', 'record_source_use', { trace_id: trace, idempotency_key: crypto.randomUUID(), uses: ids.flatMap(source_id => ['selected', 'validated', 'used'].map(stage => ({ source_id, stage }))) }); +} +function confirm(m: MemoryStore, trace: string, key = crypto.randomUUID()) { return m.mutate('github:9001', 'record_outcome', { trace_id: trace, idempotency_key: key, outcome: 'confirmed', source_ids: ['s:synthetic'], reason: 'Synthetic source verified' }); } +describe('private SQLite memory lifecycle', () => { + it('persists UTC zero-hit/scan/fetch traces, bounded pages, isolation and safe metadata', async () => fixture(m => { + const zero = save(m, 'github:9001', [], 'search'); expect(zero.timestamp).toBe('2026-01-01T00:00:00.000Z'); + const scan = save(m, 'github:9001', [source()], 'scan'); save(m, 'github:9001', [source()], 'fetch'); save(m, 'github:9002'); + const list: any = m.history('github:9001', { limit: 1 }); expect(list.traces).toHaveLength(1); expect(list.next_cursor).toBeTruthy(); + expect((m.history('github:9001', { limit: 1, cursor: list.next_cursor }) as any).traces[0].trace_id).toBe(scan.trace_id); + expect(() => m.history('github:9002', { trace_id: zero.trace_id })).toThrow('Unknown trace'); + expect(m.history('github:9001', { trace_id: zero.trace_id }).sources).toEqual([]); + expect(() => m.history('github:9002', { cursor: scan.trace_id })).toThrow('Unknown cursor'); + expect(() => m.history('github:9001', { limit: 51 })).toThrow(); + })); + it('stage order, atomic batch rollback, exact retries, conflicts and replay never add activation', async () => fixture(m => { + const t = save(m); const args = { trace_id: t.trace_id, idempotency_key: 'use-1', uses: [{ source_id: 's:synthetic', stage: 'selected' }] }; + const receipt = m.mutate('github:9001', 'record_source_use', args); expect(m.mutate('github:9001', 'record_source_use', args)).toEqual(receipt); + expect(() => m.mutate('github:9001', 'record_source_use', { ...args, uses: [{ source_id: 's:synthetic', stage: 'validated' }] })).toThrow('Idempotency conflict'); + expect(() => m.mutate('github:9001', 'record_source_use', { ...args, idempotency_key: 'bad-batch', uses: [{ source_id: 's:synthetic', stage: 'validated' }, { source_id: 'missing', stage: 'selected' }] })).toThrow(); + let h: any = m.history('github:9001', { trace_id: t.trace_id }); expect(h.sources[0].stage).toBe('selected'); expect(h.sources[0].activation.usage).toBeCloseTo(0.2); + expect(() => confirm(m, t.trace_id)).toThrow('requires used'); + use(m, t.trace_id); const before = (m.history('github:9001', { trace_id: t.trace_id }) as any).sources[0].activation.usage; + use(m, t.trace_id); expect((m.history('github:9001', { trace_id: t.trace_id }) as any).sources[0].activation.usage).toBe(before); + expect(m.strengths('github:9001', [path])).toEqual([0]); + expect(() => m.mutate('github:9002', 'record_source_use', args)).toThrow('Unknown trace'); + })); + it('half-life, caps, backwards clock and independent activation channels', async () => fixture((m, advance) => { + const t = save(m); use(m, t.trace_id); advance(3600000); + let a = (m.history('github:9001', { trace_id: t.trace_id }) as any).sources[0].activation; + expect(a.retrieved).toBeCloseTo(0.05); expect(a.usage).toBeCloseTo(0.85); + advance(-3600001); a = (m.history('github:9001', { trace_id: t.trace_id }) as any).sources[0].activation; expect(a.retrieved).toBeCloseTo(0.1); + for (let i = 0; i < 110; i++) { const next = save(m); use(m, next.trace_id); } + a = (m.history('github:9001', { trace_id: t.trace_id }) as any).sources[0].activation; expect(a.retrieved).toBe(10); expect(a.usage).toBe(10); expect(Number.isFinite(a.total)).toBe(true); + expect(decay(10, 100, 99)).toBe(10); + })); + it('real two-edge saved path, independent confirmation, duplicates, precise interleaved correction and audit', async () => fixture((m, advance, sql) => { + const first = save(m); use(m, first.trace_id); const c1 = confirm(m, first.trace_id, 'confirm-1'); + expect(c1.changes).toHaveLength(2); expect(c1.saved_sources[0].path).toEqual(path); expect(m.strengths('github:9001', [path])).toEqual([0.5]); + expect(confirm(m, first.trace_id, 'confirm-1')).toEqual(c1); confirm(m, first.trace_id, 'duplicate-confirm'); expect(m.strengths('github:9001', [path])).toEqual([0.5]); + advance(3600000); const second = save(m); use(m, second.trace_id); confirm(m, second.trace_id); expect(m.strengths('github:9001', [path])).toEqual([0.75]); + const reverse = { trace_id: first.trace_id, idempotency_key: 'reverse-1', outcome: 'corrected', confirmation_id: c1.receipt_id, reason: 'Synthetic evidence corrected' }; + const receipt = m.mutate('github:9001', 'record_outcome', reverse); expect(m.strengths('github:9001', [path])).toEqual([0.5]); + expect(m.mutate('github:9001', 'record_outcome', reverse)).toEqual(receipt); expect(receipt.changes).toHaveLength(2); + expect(sql.exec('SELECT edge FROM memory_edges').toArray().map(x => x.edge).sort()).toEqual(path.map(edgeKey).sort()); + const h: any = m.history('github:9001', { trace_id: first.trace_id, limit: 2 }); expect(h.next_cursor).toBeTruthy(); + const audit = (m.history('github:9001', { trace_id: first.trace_id, limit: 50 }) as any).audit; expect(audit.find((r: any) => r.receipt_id === c1.receipt_id).active_credits.every((x: any) => x.active === 0)).toBe(true); + expect(() => m.mutate('github:9001', 'record_outcome', { ...reverse, trace_id: second.trace_id, idempotency_key: 'wrong-trace' })).toThrow('Unknown confirmation'); + expect(() => m.mutate('github:9001', 'record_outcome', { ...reverse, confirmation_id: undefined, idempotency_key: 'ambiguous' })).toThrow('requires confirmation_id'); + })); + it('never creates lexical/zero-hop edges; capped credit reverses only its applied delta', async () => fixture(m => { + const lexical = { ...source(), path: [] }; const t = save(m, 'github:9001', [lexical]); use(m, t.trace_id); expect(confirm(m, t.trace_id).changes).toEqual([]); + let last: any; let lastTrace = ''; + for (let i = 0; i < 21; i++) { const next = save(m); use(m, next.trace_id); last = confirm(m, next.trace_id); lastTrace = next.trace_id; } + expect(m.strengths('github:9001', [path])).toEqual([10]); expect(last.changes.every((c: any) => c.delta === 0)).toBe(true); + m.mutate('github:9001', 'record_outcome', { trace_id: lastTrace, idempotency_key: 'capped-reversal', outcome: 'rolled_back', confirmation_id: last.receipt_id, reason: 'Synthetic rollback' }); expect(m.strengths('github:9001', [path])).toEqual([10]); + })); + it('concurrent requests serialize and share same-principal receipts across clients', async () => { + const stub = env.ISSUE_STORE.get(env.ISSUE_STORE.idFromName(crypto.randomUUID())); + const call = async (action: string, args: unknown, principal = 'github:9001') => { const res = await stub.fetch(new Request('http://store/memory', { method: 'POST', body: JSON.stringify({ principal, action, args }) })); return { status: res.status, body: await res.json() as any }; }; + const { body: t } = await call('save', { request: { mode: 'search' }, settings: {}, sources: [source()] }); + const args = { trace_id: t.trace_id, idempotency_key: 'simultaneous', uses: ['selected','validated','used'].map(stage => ({ source_id: 's:synthetic', stage })) }; + const results = await Promise.all(Array.from({ length: 8 }, () => call('record_source_use', args))); expect(new Set(results.map(r => r.body.receipt_id)).size).toBe(1); + const h = await call('memory_history', { trace_id: t.trace_id }); expect(h.body.sources[0].activation.usage).toBeCloseTo(1.7); expect(h.body.audit).toHaveLength(1); + expect((await call('memory_history', { trace_id: t.trace_id }, 'github:9002')).status).toBe(400); + }); + it('save failures roll back trace and activation together', async () => fixture((m, _advance, sql) => { + sql.exec("CREATE TRIGGER synthetic_save_failure BEFORE INSERT ON memory_activation BEGIN SELECT RAISE(ABORT, 'synthetic save failure'); END;"); + expect(() => save(m)).toThrow('synthetic save failure'); + expect(sql.exec('SELECT COUNT(*) AS n FROM memory_traces').one().n).toBe(0); + expect(sql.exec('SELECT COUNT(*) AS n FROM memory_activation').one().n).toBe(0); + })); + +}); diff --git a/src/pipeline/embed-comment.ts b/src/pipeline/embed-comment.ts index a66bfb7..a4806c4 100644 --- a/src/pipeline/embed-comment.ts +++ b/src/pipeline/embed-comment.ts @@ -13,7 +13,7 @@ import type { PRReviewRecord, PRReviewCommentRecord, } from "../types.js"; -import { upsertFtsRow } from "../fts.js"; +import { upsertFtsRow, repairSourceEventIdentity } from "../fts.js"; import { isBotSender, isBodyTooShort } from "./ingest-filter.js"; import { computeBodyHash, prepareCommentEmbeddingInput } from "./hash.js"; import { generateEmbedding } from "./embedding.js"; @@ -94,6 +94,7 @@ export async function ingestIssueComment( if (existingResp.ok) { const existing = (await existingResp.json()) as IssueCommentRecord; if (existing.bodyHash === bodyHash) { + await repairSourceEventIdentity(env.DB_FTS, await issueCommentVectorId(repo, comment.id), "comment", comment.id); return { embedded: false, skippedUnchanged: true, filtered: false, failed: false }; } } @@ -125,6 +126,7 @@ export async function ingestIssueComment( vectorId: vid, repo, type: "issue_comment", + commentId: comment.id, state: "active", labels: "", milestone: "", @@ -205,6 +207,7 @@ export async function ingestPRReview( if (existingResp.ok) { const existing = (await existingResp.json()) as PRReviewRecord; if (existing.bodyHash === bodyHash) { + await repairSourceEventIdentity(env.DB_FTS, await prReviewVectorId(repo, review.id), "review", review.id); return { embedded: false, skippedUnchanged: true, filtered: false, failed: false }; } } @@ -238,6 +241,7 @@ export async function ingestPRReview( vectorId: vid, repo, type: "pr_review", + reviewId: review.id, state: review.state, labels: "", milestone: "", @@ -327,6 +331,7 @@ export async function ingestPRReviewComment( if (existingResp.ok) { const existing = (await existingResp.json()) as PRReviewCommentRecord; if (existing.bodyHash === bodyHash) { + await repairSourceEventIdentity(env.DB_FTS, await prReviewCommentVectorId(repo, comment.id), "comment", comment.id); return { embedded: false, skippedUnchanged: true, filtered: false, failed: false }; } } @@ -361,6 +366,7 @@ export async function ingestPRReviewComment( vectorId: vid, repo, type: "pr_review_comment", + commentId: comment.id, state: "active", labels: "", milestone: "", diff --git a/src/rerank.ts b/src/rerank.ts index c8ffb8a..b0f6018 100644 --- a/src/rerank.ts +++ b/src/rerank.ts @@ -180,10 +180,7 @@ export async function rerankCandidates( top_k: trimmedCandidates.length, })) as unknown as RerankerResponse; } catch (err) { - console.error( - "rerankCandidates: bge-reranker-base call failed:", - err instanceof Error ? err.message : String(err), - ); + console.error("rerankCandidates: bge-reranker-base call failed:"); return null; } diff --git a/src/scan.ts b/src/scan.ts index 8dc1623..67069eb 100644 --- a/src/scan.ts +++ b/src/scan.ts @@ -93,6 +93,7 @@ export interface ScanParams { } export interface ScanOutcome { + incomplete?: boolean; rows: ScanRow[]; /** Window floor actually applied (the caller's `since`, or the default). */ since: string; @@ -155,6 +156,7 @@ export async function runScan( const rows: ScanRow[] = []; let truncated = false; + let incomplete = false; const collect = async ( path: string, map: (record: T) => ScanRow | null, @@ -164,7 +166,7 @@ export async function runScan( const res = await store.fetch( new Request(`http://store/${path}?${buildParams(pathPrefix).toString()}`), ); - if (!res.ok) return; + if (!res.ok) throw new Error("Scan source unavailable"); const records = (await res.json()) as T[]; // A full page means the cap, not the window, ended the read. if (records.length >= storeLimit) truncated = true; @@ -173,6 +175,7 @@ export async function runScan( if (row) rows.push(row); } } catch { + incomplete = true; // Non-critical; continue with the other sources. } }; @@ -365,5 +368,5 @@ export async function runScan( if (filtered.length > topK) truncated = true; - return { rows: filtered.slice(0, topK), since, truncated }; + return { rows: filtered.slice(0, topK), since, truncated, incomplete }; } diff --git a/src/search-contract.ts b/src/search-contract.ts new file mode 100644 index 0000000..b7d07a6 --- /dev/null +++ b/src/search-contract.ts @@ -0,0 +1,175 @@ +import { z } from "zod"; +export const INCLUDE_CONTENT_MAX_DOCS = 5; +export const searchInputSchema = z.object({ + query: z + .string().max(4096) + .optional() + .describe( + "Natural language search query. When omitted or empty, the tool " + + "switches to metadata-only scan mode and results are ordered " + + "by the timestamp implied by sort (default sort=\"updated_desc\" for empty query).", + ), + repo: z + .string().max(256) + .optional() + .describe( + "Filter by repository — full slug (owner/repo), exact match. " + + "A bare repository name (\"my-repo\") matches nothing and yields an empty result set; " + + "search mode flags that case as \"repo\" in the response's filters_unmatched.", + ), + path_prefix: z + .string().max(256) + .optional() + .describe( + "Filter repository docs by a repository-relative directory prefix before ranking. " + + "Requires type=\"doc\", a trailing /, no leading /, backslash, NUL, empty, . or .. path segments, " + + "and at most 64 UTF-8 bytes. Search mode reports an unmatched value as \"path_prefix\" in filters_unmatched.", + ), + state: z + .enum(["open", "closed", "all"]) + .optional() + .default("all") + .describe("Filter by state"), + labels: z + .array(z.string().max(256)).max(50) + .optional() + .describe("Filter by label names (AND logic)"), + milestone: z + .string().max(256) + .optional() + .describe("Filter by milestone title"), + assignee: z + .string().max(256) + .optional() + .describe("Filter by assignee login"), + type: z + .enum([ + "issue", + "pull_request", + "release", + "doc", + "wiki_doc", + "diff", + "issue_comment", + "pr_review", + "pr_review_comment", + "all", + ]) + .optional() + .default("all") + .describe( + "Filter by type (default: all). " + + "\"doc\" = repository docs (files in /docs/ etc.). " + + "\"wiki_doc\" = GitHub Wiki pages (separate from repo docs; both surfaces co-exist). " + + "\"diff\" = per-file commit diffs. " + + "\"issue_comment\" = top-level comments on issues and PRs. " + + "\"pr_review\" = PR review bodies (approve / request_changes / comment). " + + "\"pr_review_comment\" = inline per-line review comments on PR diffs.", + ), + top_k: z + .number().int() + .min(1) + .max(50) + .optional() + .default(10) + .describe("Max results (default: 10, max: 50)"), + fusion: z + .enum(["rrf", "dense_only", "sparse_only"]) + .optional() + .default("rrf") + .describe( + "Fusion strategy. Default: rrf (Reciprocal Rank Fusion over dense + sparse). " + + "dense_only = Vectorize only. sparse_only = D1 FTS5 BM25 only. " + + "Use rrf unless debugging a specific ranker. Ignored in metadata-only scan mode (empty query).", + ), + rerank: z + .boolean() + .optional() + .default(true) + .describe( + "Cross-encoder reranking with @cf/baai/bge-reranker-base. Default: true. " + + "When enabled, the fused (or single-ranker) candidates are overfetched (top_k × 5, max 50), " + + "post-filtered, then re-scored by the cross-encoder before being trimmed to top_k. " + + "Set false to disable (faster, no Workers AI rerank cost; recommended for debugging or " + + "when query is a short identifier where lexical match is already decisive). " + + "Ignored in metadata-only scan mode (empty query).", + ), + sort: z + .enum(["relevance", "updated_desc", "created_desc"]) + .optional() + .describe( + "Result ordering. Default: \"relevance\" when query is non-empty, \"updated_desc\" when query is empty. " + + "\"updated_desc\" / \"created_desc\" force time-ordered output and override ranker scores.", + ), + since: z + .string().max(256) + .optional() + .describe( + "ISO 8601 timestamp (inclusive) — keep only results whose updated_at >= since. " + + "Pair with sort=\"updated_desc\" + empty query for an activity scan. " + + "Default in scan mode: 7 days back from until (or from now when until is omitted).", + ), + until: z + .string().max(256) + .optional() + .describe( + "ISO 8601 timestamp (exclusive) — keep only results whose updated_at < until. " + + "In scan mode the [since, until) window is applied inside the index, so any window " + + "holding rows returns rows however far back it sits; the response carries " + + "truncated: true when the window holds more than one page.", + ), + include_content: z + .boolean() + .optional() + .default(false) + .describe( + "When true, inline live doc bodies from the GitHub contents API and wiki_doc bodies from raw wiki URLs. " + + "Successful live bodies carry content_source=github_live and content_version=SHA-256 of the returned text; " + + "their source_id identifies this live version, separately from the indexed snapshot. " + + `Capped at ${INCLUDE_CONTENT_MAX_DOCS} rows of each type. Non-doc rows are unaffected. ` + + "Graph-axis bodies remain the indexed copy, marked content_source=index." + ), + vector_ids: z + .array(z.string().max(256)).max(50) + .max(50) + .optional() + .describe( + "Stored-content fetch. Pass the vector_id values carried by earlier search-mode results " + + "(scan-mode rows come from the structured store and carry none) to read back the " + + "body text the index holds for those exact rows, for every type — issue, pull_request, " + + "issue_comment, pr_review, pr_review_comment, release, diff, doc, wiki_doc. " + + "Served from D1: no GitHub API call is made. Takes precedence over the other modes — query, " + + "sort, and every metadata filter are ignored when this is present, because the rows are named " + + "rather than selected. " + + "The text is the INDEXED copy of the body (the embedding input), truncated at " + + `${8000} characters — not the live source. Each row carries content_truncated ` + + "so a prefix is never mistaken for a whole body, and the response carries content_source: \"index\". " + + "Unknown or stale ids are listed in not_found and the remaining rows still return. " + + `Max ${50} ids per call. ` + + "Treat vector_id as a handle for a row you just found, not a durable identifier: the id scheme " + + "has been migrated before and may be again, so do not store one for later use.", + ), + graph_expand: z + .boolean() + .optional() + .default(false) + .describe( + "Opt-in GraphRAG expansion (search mode only). When true, after fusion the top " + + "results seed a traversal of the Decision-Structure mention graph (D1 doc_edges); " + + "related wiki pages are returned in a separate graph_results array marked with " + + "graph_hop / graph_from — never mixed into results, and carrying no score. " + + "Default false preserves standard hybrid ranking (no graph read, " + + "no graph_results field).", + ), + use_memory: z.boolean().optional().default(false).describe("Order graph candidates within equal hops by your learned strength; keyword ranks and scores unchanged."), + graph_hops: z + .number().int() + .min(1) + .max(2) + .optional() + .default(1) + .describe( + "Graph traversal depth for graph_expand (1 or 2). Default 1. Ignored when graph_expand is false.", + ), + }); +export const searchOutputSchema = z.object({ count: z.number().int().nonnegative(), mode: z.enum(['search','scan','fetch']), results: z.array(z.record(z.string(), z.unknown())).max(50), graph_results: z.array(z.record(z.string(), z.unknown())).max(30).optional(), trace_id: z.string().optional(), timestamp: z.string().optional(), memory_policy: z.record(z.string(), z.unknown()).optional(), memory_unavailable: z.boolean().optional(), feedback_available: z.boolean() }).passthrough(); diff --git a/src/store.ts b/src/store.ts index e850bdc..10f70b5 100644 --- a/src/store.ts +++ b/src/store.ts @@ -18,6 +18,7 @@ import type { PRReviewCommentRecord, PollWatermark, } from "./types.js"; +import { MemoryStore } from "./memory.js"; import { pathPrefixRange } from "./path-prefix.js"; /** @@ -249,10 +250,12 @@ function rowToDiffRecord(row: DiffRow): DiffRecord { export class IssueStore implements DurableObject { private sql: SqlStorage; + private memory: MemoryStore; constructor(state: DurableObjectState, _env: Env) { this.sql = state.storage.sql; this.initSchema(); + this.memory = new MemoryStore(this.sql, fn => state.storage.transactionSync(fn)); } /** @@ -1097,6 +1100,39 @@ export class IssueStore implements DurableObject { async fetch(request: Request): Promise { const url = new URL(request.url); const path = url.pathname; + if (request.method === "GET" && path === "/source-identities") { + const repo = url.searchParams.get("repo") ?? ""; + const limit = Number(url.searchParams.get("limit") ?? 50); + const cursor = url.searchParams.get("cursor") ?? ""; + const match = cursor ? /^(issue_comment|pr_review|pr_review_comment):([0-9]+)$/.exec(cursor) : null; + if (!repo || repo.length > 256 || !Number.isInteger(limit) || limit < 1 || limit > 100 || (cursor && !match)) return new Response("Invalid identity page", { status: 400 }); + const definitions = [{ type: "issue_comment", table: "issue_comments", id: "comment_id" }, { type: "pr_review", table: "pr_reviews", id: "review_id" }, { type: "pr_review_comment", table: "pr_review_comments", id: "comment_id" }]; + const rows: { repo: string; type: string; event_id: number }[] = []; + for (const d of definitions) { + if (match && d.type < match[1]) continue; + const after = match && d.type === match[1] ? Number(match[2]) : 0; + for (const r of this.sql.exec(`SELECT ${d.id} AS event_id FROM ${d.table} WHERE repo=? AND ${d.id}>? ORDER BY ${d.id} LIMIT ?`, repo, after, limit + 1).toArray()) rows.push({ repo, type: d.type, event_id: Number(r.event_id) }); + } + rows.sort((a,b) => a.type.localeCompare(b.type) || a.event_id - b.event_id); + const page = rows.slice(0, limit); const last = page.at(-1); + return Response.json({ rows: page, next_cursor: rows.length > limit && last ? `${last.type}:${last.event_id}` : null, done: rows.length <= limit }); + } + if (request.method === "POST" && path === "/memory") { + try { + const body = await request.json() as { principal: string; action: string; args: unknown }; + if (!/^github:[1-9][0-9]*$/.test(body.principal)) throw new Error("Invalid authenticated principal"); + const data = body.action === "save" ? this.memory.save(body.principal, body.args as Parameters[1]) + : body.action === "memory_history" ? this.memory.history(body.principal, body.args) + : body.action === "strengths" ? this.memory.strengths(body.principal, body.args as Parameters[1]) + : body.action === "record_source_use" || body.action === "record_outcome" ? this.memory.mutate(body.principal, body.action, body.args) + : (() => { throw new Error("Unknown memory action"); })(); + return Response.json(data); + } catch (err) { + const codes: Record = { 'Unknown trace': 'unknown_trace', 'Unknown source in trace': 'unknown_source', 'Unknown cursor': 'unknown_cursor', 'Idempotency conflict': 'idempotency_conflict', 'Usage stage order violation': 'stage_order_violation', 'Confirmation requires used source': 'used_source_required', 'Unknown confirmation for trace': 'unknown_confirmation', 'Reversal requires confirmation_id only': 'confirmation_id_required', 'confirmed requires source_ids only': 'source_ids_required' }; + const code = codes[err instanceof Error ? err.message : ''] ?? 'invalid_memory_request'; + return Response.json({ error: code }, { status: 400 }); + } + } /** * Query-string form of the recency window shared by every `/recent*`