diff --git a/.github/workflows/skills.yml b/.github/workflows/skills.yml new file mode 100644 index 0000000..21f6458 --- /dev/null +++ b/.github/workflows/skills.yml @@ -0,0 +1,24 @@ +name: Skills + +on: + push: + branches: [main] + pull_request: + +jobs: + verify-skills: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + + - run: python -m pip install --upgrade pip pyyaml + + - run: bash skills/scripts/verify.sh + + - run: bash skills/scripts/test-install.sh + + - run: git diff --check diff --git a/README.md b/README.md index 3c6c605..95b9311 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,8 @@ Claude Code and Codex can consume directly. The goal is simple: make handoff work cheaper. A PRD should become a usable HTML mock. A design screenshot should become a component decomposition plan. An OpenAPI file should become a typed client scaffold. A project should carry its -own context so the next agent run does not start from zero. +own context, source-cited memory, and review artifacts so the next agent run +does not start from zero. ## What Knowject Skills Do @@ -48,6 +49,9 @@ own context so the next agent run does not start from zero. | UI screenshot, PDF page, or Figma export | `knowject-read-design` | Component decomposition plan and skeleton files | | Express routes or OpenAPI document | `knowject-read-api` | Endpoint inventory and typed-client scaffold | | OpenAPI 3 document | `knowject-api-to-types` | TypeScript response types wired into the generated client | +| Project artifacts, handoff notes, diffs, or decisions | `knowject-memory-capture` | Source-cited durable project memory under `knowject/memory/` | +| Project knowledge sources | `knowject-rag-eval` | Source-cited RAG and citation eval cases under `knowject/evals/` | +| API surfaces or endpoint inventories | `knowject-mcp-tool-designer` | MCP tool design artifacts under `knowject/tools/` | These Skills are intentionally not generic brainstorming or code-review prompts. Each one is scoped to a cross-role handoff where project context @@ -57,8 +61,10 @@ matters. 1. Run `knowject-context-init` once in your project. 2. Commit `knowject/context.yaml` as the project anchor. -3. Use the other Skills to turn PRDs, UI mocks, and API specs into concrete - artifacts. +3. Optionally run `knowject-memory-capture` to preserve source-cited project + facts, decisions, workflows, risks, and lessons under `knowject/memory/`. +4. Use the other Skills to turn PRDs, UI mocks, API specs, knowledge sources, + and endpoint inventories into concrete review artifacts. `knowject/context.yaml` is the shared contract. It records the project type, frontend and backend stack, design source paths, API source paths, client output @@ -70,6 +76,18 @@ your-project/ knowject/ context.yaml README.md + memory/ + README.md + project-memory.yaml + evals/ + README.md + rag-eval-cases.yaml + rag-eval-report.md + tools/ + README.md + mcp-tools.plan.md + mcp-tools.schema.json + tool-risk-report.md ``` ## Install @@ -109,7 +127,7 @@ pip install pyyaml bash skills/scripts/verify.sh ``` -The installer symlinks every `knowject-*` Skill into: +The installer symlinks every `knowject-*` Skill plus the shared support folder `_shared` into: ```text ~/.claude/skills/ @@ -139,6 +157,9 @@ After that, use the Skills directly: /knowject design /knowject api /knowject types +/knowject memory +/knowject rag eval +/knowject mcp tools ``` Natural-language requests work too, as long as the intent matches the Skill: @@ -148,6 +169,9 @@ Natural-language requests work too, as long as the intent matches the Skill: 把这张设计稿拆成 React + Antd 组件骨架。 找一下用户列表 endpoint,并生成 typed client。 把这个 OpenAPI 文档转成 TypeScript response types。 +把这次交接总结成项目记忆。 +基于这些知识库文档生成 RAG 引用评测用例。 +把这些 endpoint 设计成带风险 gate 的 MCP tools。 ``` ## Skill Boundaries @@ -159,6 +183,9 @@ Natural-language requests work too, as long as the intent matches the Skill: | `knowject-read-design` | UI decomposition and component skeletons | Does not wire state, routes, tests, or real data | | `knowject-read-api` | API discovery and typed-client scaffold | Phase 2 supports Express and OpenAPI sources | | `knowject-api-to-types` | OpenAPI 3 response types for generated clients | Day-1 scope is response types, not full request typing | +| `knowject-memory-capture` | Source-cited file-based project memory | No DB, vector store, platform UI, daemon, secrets, or uncited facts | +| `knowject-rag-eval` | Source-cited RAG and citation eval artifacts | No live model calls, Chroma, MongoDB, platform UI, network scoring, or external eval dependency | +| `knowject-mcp-tool-designer` | API-to-MCP tool design with risk labels and confirmation gates | Does not implement an MCP server, runtime handlers, auth code, API routes, or UI | This makes the output reviewable. A Skill should produce a useful first pass, not silently mutate a production codebase. @@ -170,9 +197,12 @@ bash skills/scripts/verify.sh bash skills/scripts/test-install.sh ``` -`verify.sh` checks the manifest, Skill frontmatter, Codex adapters, -`context.yaml` examples, route extractors, OpenAPI extractors, brand extraction, -framework extraction, type extraction, and client rewrite fixtures. +`verify.sh` checks the manifest, Skill frontmatter, Codex adapters, per-Skill +README coverage, `context.yaml` examples, route extractors, OpenAPI extractors, +typed-client generation, brand extraction, framework extraction, type +extraction, client rewrite fixtures, the memory validator, RAG eval validator, +and MCP tool schema fixture. `test-install.sh` checks installer idempotency and +installed shared-file references. ## The Platform Behind the Skills diff --git a/README.zh-CN.md b/README.zh-CN.md index 631336a..441c176 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -14,7 +14,7 @@ Knowject 现在以 Skill 为核心。 它是一个面向 Claude Code 与 Codex 的项目级 Skill 包,也是一套围绕项目知识持续演进的 AI 工作台。第一可用入口是 [`skills/`](./skills/README.md):一组可以安装到本机 agent 工具里的 Knowject Skills,用真实项目上下文驱动 PRD、设计稿、API 文档与代码交接。 -核心目标很直接:降低跨角色交接成本。PRD 不应该停在文字里,设计稿不应该只靠人工拆组件,OpenAPI 不应该还要手写 `unknown` client。Knowject Skills 让这些输入变成可检查、可继续改的工程初稿。 +核心目标很直接:降低跨角色交接成本。PRD 不应该停在文字里,设计稿不应该只靠人工拆组件,OpenAPI 不应该还要手写 `unknown` client,项目经验也不应该散落在临时对话里。Knowject Skills 让这些输入变成可检查、可继续改的工程初稿、带来源的项目记忆和评审产物。 ## Knowject Skills 能做什么 @@ -25,6 +25,9 @@ Knowject 现在以 Skill 为核心。 | UI 截图、PDF 页面或 Figma 导出 | `knowject-read-design` | 组件拆解计划与组件骨架文件 | | Express 路由或 OpenAPI 文档 | `knowject-read-api` | Endpoint 清单与 typed-client 初稿 | | OpenAPI 3 文档 | `knowject-api-to-types` | 写入生成 client 的 TypeScript response types | +| 项目文档、交接记录、diff 或决策 | `knowject-memory-capture` | 写入 `knowject/memory/` 的带来源项目记忆 | +| 项目知识源 | `knowject-rag-eval` | 写入 `knowject/evals/` 的带来源 RAG / 引用评测用例 | +| API 表面或 endpoint 清单 | `knowject-mcp-tool-designer` | 写入 `knowject/tools/` 的 MCP tool 设计产物 | 这些不是泛泛的头脑风暴或代码审查 prompt。每个 Skill 都对应一个跨角色交接场景,并且依赖项目自己的 `knowject/context.yaml`。 @@ -32,7 +35,8 @@ Knowject 现在以 Skill 为核心。 1. 在目标项目里运行一次 `knowject-context-init`。 2. 提交 `knowject/context.yaml`,把它作为项目上下文锚点。 -3. 用其他 Knowject Skills 把 PRD、设计稿、API 文档转成可继续交付的产物。 +3. 可选运行 `knowject-memory-capture`,把项目事实、决策、流程、风险和经验写成带来源的 `knowject/memory/` 文件。 +4. 用其他 Knowject Skills 把 PRD、设计稿、API 文档、知识源和 endpoint 清单转成可继续交付的评审产物。 `knowject/context.yaml` 是共享合同。它记录项目类型、前后端技术栈、设计稿路径、API 源路径、client 输出目录与品牌 token。它不应该包含 secrets、base URL、API key 或环境配置。 @@ -41,6 +45,18 @@ your-project/ knowject/ context.yaml README.md + memory/ + README.md + project-memory.yaml + evals/ + README.md + rag-eval-cases.yaml + rag-eval-report.md + tools/ + README.md + mcp-tools.plan.md + mcp-tools.schema.json + tool-risk-report.md ``` ## 安装 @@ -77,7 +93,7 @@ pip install pyyaml bash skills/scripts/verify.sh ``` -安装脚本会把所有 `knowject-*` Skill 软链接到: +安装脚本会把所有 `knowject-*` Skill 和共享支持目录 `_shared` 软链接到: ```text ~/.claude/skills/ @@ -107,6 +123,9 @@ bash skills/scripts/verify.sh /knowject design /knowject api /knowject types +/knowject memory +/knowject rag eval +/knowject mcp tools ``` 自然语言也可以触发对应 Skill: @@ -116,6 +135,9 @@ bash skills/scripts/verify.sh 把这张设计稿拆成 React + Antd 组件骨架。 找一下用户列表 endpoint,并生成 typed client。 把这个 OpenAPI 文档转成 TypeScript response types。 +把这次交接总结成项目记忆。 +基于这些知识库文档生成 RAG 引用评测用例。 +把这些 endpoint 设计成带风险 gate 的 MCP tools。 ``` ## Skill 边界 @@ -127,6 +149,9 @@ bash skills/scripts/verify.sh | `knowject-read-design` | UI 拆解与组件骨架 | 不接状态、路由、测试或真实数据 | | `knowject-read-api` | API 发现与 typed-client 初稿 | Phase 2 仅支持 Express 与 OpenAPI | | `knowject-api-to-types` | 为生成的 client 补 OpenAPI 3 response types | Day-1 只处理 response type,不做完整 request typing | +| `knowject-memory-capture` | 带来源的文件型项目记忆 | 不做 DB、向量库、平台 UI、daemon,不写 secrets 或无来源事实 | +| `knowject-rag-eval` | 带来源的 RAG / 引用评测产物 | 不做 live model call、Chroma、MongoDB、平台 UI、联网评分或外部 eval 依赖 | +| `knowject-mcp-tool-designer` | 带风险标签和确认 gate 的 API-to-MCP tool 设计 | 不实现 MCP server、runtime handler、auth 代码、API 路由或 UI | 这个边界是故意的。Knowject Skills 应该产出有用的一版初稿,而不是悄悄改动生产代码。 @@ -137,7 +162,7 @@ bash skills/scripts/verify.sh bash skills/scripts/test-install.sh ``` -`verify.sh` 会检查 manifest、Skill frontmatter、Codex adapter、`context.yaml` 示例、路由提取器、OpenAPI 提取器、品牌提取、框架提取、类型提取与 client rewrite fixtures。 +`verify.sh` 会检查 manifest、Skill frontmatter、Codex adapter、各 Skill README 覆盖、`context.yaml` 示例、路由提取器、OpenAPI 提取器、typed-client 生成、品牌提取、框架提取、类型提取、client rewrite fixtures、memory validator、RAG eval validator 与 MCP tool schema fixture。`test-install.sh` 会检查安装幂等性和安装后的共享文件相对路径。 ## Skill 背后的平台 diff --git a/skills/README.md b/skills/README.md index 64951c8..82f70d2 100644 --- a/skills/README.md +++ b/skills/README.md @@ -2,7 +2,7 @@ Cross-role collaboration Skills for Claude Code and Codex. Project-anchored. Opinionated. -> Status: Phase 4 Day-1 complete. Shipped: `knowject-context-init`, `knowject-read-api`, `knowject-prd-to-mock`, `knowject-read-design`, `knowject-api-to-types`, `knowject-memory-capture`. Next: Claude Code plugin marketplace. +> Status: Phase 6 Day-1 complete. Shipped: `knowject-context-init`, `knowject-read-api`, `knowject-prd-to-mock`, `knowject-read-design`, `knowject-api-to-types`, `knowject-memory-capture`, `knowject-rag-eval`, `knowject-mcp-tool-designer`. Next: Claude Code plugin marketplace. --- @@ -18,11 +18,17 @@ Skill catalog: | `knowject-api-to-types` | Generate TypeScript response types from OpenAPI and wire them into generated clients. | [`README`](./knowject-api-to-types/README.md) | | `knowject-read-design` | Turn UI design sources into component decomposition plans and skeleton code. | [`README`](./knowject-read-design/README.md) | | `knowject-memory-capture` | Capture source-cited durable project memory from artifacts, handoffs, diffs, and decisions. | [`README`](./knowject-memory-capture/README.md) | +| `knowject-rag-eval` | Generate source-cited RAG and citation evaluation cases and reports. | [`README`](./knowject-rag-eval/README.md) | +| `knowject-mcp-tool-designer` | Design API-to-MCP tool candidates with risk labels and confirmation gates. | [`README`](./knowject-mcp-tool-designer/README.md) | Each Skill reads `knowject/context.yaml` in your project to ground its output in your brand, your tech stack, and your conventions instead of producing generic AI output. `knowject-memory-capture` turns project artifacts, handoff notes, diffs, and decisions into source-cited durable memory under `knowject/memory/`, so Claude Code and Codex can reuse stable context across sessions without guessing. +`knowject-rag-eval` turns project knowledge sources into reviewable eval cases under `knowject/evals/` for retrieval, citation support, unsupported claims, and fact-vs-plan conflicts. + +`knowject-mcp-tool-designer` maps API surfaces into reviewable MCP tool design artifacts under `knowject/tools/`, including input schema, risk labels, confirmation gates, auth notes, audit notes, and rollback notes. + This is not a new agent CLI. It is a pack of Skills that Claude Code and Codex consume directly, similar to the `longbridge` or `superpowers` skill packs. --- @@ -65,7 +71,7 @@ cd knowject bash skills/scripts/install.sh ``` -`install.sh` symlinks every `knowject-*` Skill into `~/.claude/skills/` and `~/.codex/skills/`. Re-run anytime to refresh. +`install.sh` symlinks every `knowject-*` Skill plus the shared support folder `_shared` into `~/.claude/skills/` and `~/.codex/skills/`. Re-run anytime to refresh. ### Fallback: copy instead of symlink @@ -127,6 +133,33 @@ your-project/ Every memory item must cite source evidence. Do not put secrets, environment values, tokens, DB URLs, cookies, auth headers, or private credentials in `knowject/memory/`. +After running `knowject-rag-eval`, the same folder can also contain file-based RAG and citation eval artifacts: + +```text +your-project/ + knowject/ + evals/ + README.md + rag-eval-cases.yaml # source-cited eval cases + rag-eval-report.md # human review report +``` + +Every eval case must cite source evidence. Day-1 evals are review artifacts: no live model calls, vector-store queries, platform API calls, or external scoring. + +After running `knowject-mcp-tool-designer`, the same folder can also contain MCP tool design artifacts: + +```text +your-project/ + knowject/ + tools/ + README.md + mcp-tools.plan.md + mcp-tools.schema.json + tool-risk-report.md +``` + +Day-1 MCP tool design is artifact-only: no MCP server, runtime handlers, platform API, UI, auth code, or external API calls. + --- ## Contribute @@ -145,6 +178,7 @@ We do not accept: - Pure template generators that do not read project context. See [`_shared/contributing-skills.md`](./_shared/contributing-skills.md) for the full PR checklist. +GitHub Actions runs `skills/scripts/verify.sh`, `skills/scripts/test-install.sh`, and `git diff --check` on pull requests and pushes to `main`. --- @@ -160,4 +194,6 @@ See [LICENSE](./LICENSE). Source-available: personal evaluation and non-commerci - [x] Phase 2: `knowject-read-api`, `knowject-prd-to-mock`, `knowject-read-design` - [x] Phase 3: `knowject-api-to-types` - [x] Phase 4: `knowject-memory-capture` +- [x] Phase 5: `knowject-rag-eval` +- [x] Phase 6: `knowject-mcp-tool-designer` - [ ] Next: Claude Code plugin manifest, official marketplace listing diff --git a/skills/ROADMAP.md b/skills/ROADMAP.md index 3b5bf10..0517a79 100644 --- a/skills/ROADMAP.md +++ b/skills/ROADMAP.md @@ -14,6 +14,8 @@ Canonical product roadmap for the `skills/` distribution. The Active catalog, de | `knowject-read-design` | Phase 2 — Tier 1 | `docs/plans/tasks-knowject-skills-read-design.md` | | `knowject-api-to-types` | Phase 3 | `docs/plans/tasks-knowject-skills-api-to-types.md` | | `knowject-memory-capture` | Phase 4 — memory | GitHub issue #1 | +| `knowject-rag-eval` | Phase 5 — RAG eval | GitHub issue #8 | +| `knowject-mcp-tool-designer` | Phase 6 — MCP tool design | GitHub issue #9 | ## Deferred (2026-05-17 narrowing) @@ -41,6 +43,64 @@ Versioning is on the whole product (`manifest.yaml#version`), not per-Skill. ## Decision log +### 2026-05-24 — Phase 6 shipped: mcp-tool-designer + +**Trigger:** Hangzhou AI Agent / AI application market scan repeatedly +mentions MCP, tool calling, function calling, OpenAPI, agent tool +ecosystems, and permission / confirmation gates. `knowject-read-api` +and `knowject-api-to-types` already discover and type API surfaces; +the next conservative step is tool design, not runtime execution. + +**Outcome:** `knowject-mcp-tool-designer` shipped as a Day-1 Skill for +mapping API endpoints into MCP tool candidates with input schema, risk +labels, confirmation gates, auth notes, audit notes, and rollback +notes. + +**Day-1 scope:** design artifacts only under `knowject/tools/`: +`README.md`, `mcp-tools.plan.md`, `mcp-tools.schema.json`, and +`tool-risk-report.md`. The Skill requires `knowject/context.yaml`, +supports OpenAPI sources, Express route sources, `knowject-read-api` +endpoint inventories, and explicit endpoint lists, labels risks as +`read_only`, `low_write`, `destructive`, `external_side_effect`, or +`sensitive_data`, and requires confirmation gates for destructive, +external, sensitive, and bulk operations. + +**Deliberate non-goals:** no MCP server, no runtime tool handlers, no +external API calls, no app route changes, no platform UI, no +auth/permission code, and no `knowject-read-api` behavior changes. + +**Resume signal:** API-to-MCP tool design with input schema, risk +labels, confirmation gates, auth notes, audit notes, and rollback +considerations. + +### 2026-05-24 — Phase 5 shipped: rag-eval + +**Trigger:** Hangzhou AI Agent / AI application market scan shows +RAG, Eval, citation traceability, knowledge bases, vector retrieval, +and verifiable output as high-signal requirements. Existing Knowject +Skills cover project anchoring, cross-role handoff, API-to-types, and +durable memory, but not lightweight RAG/citation quality evaluation. + +**Outcome:** `knowject-rag-eval` shipped as a Day-1 Skill for creating +reviewable RAG and citation evaluation artifacts from project knowledge +sources. + +**Day-1 scope:** file-based, source-cited eval artifacts under +`knowject/evals/`: `README.md`, `rag-eval-cases.yaml`, and +`rag-eval-report.md`. The Skill requires `knowject/context.yaml`, +requires source evidence for every eval case, documents source recall, +citation support, unsupported-claim, fact-vs-plan, and conflict +resolution eval types, avoids secrets and environment values, and uses +a diff-confirm write gate. + +**Deliberate non-goals:** no live model calls, no Chroma queries, no +MongoDB / API / UI changes, no RAGAS or external eval dependency, no +automated scoring against a running chatbot, and no CI that requires +network. + +**Resume signal:** source-cited RAG evaluation, citation support checks, +unsupported-claim review, and fact-vs-roadmap conflict cases. + ### 2026-05-24 — Phase 4 shipped: memory-capture **Trigger:** Hangzhou AI Agent / AI application market scan shows diff --git a/skills/install.md b/skills/install.md index 62a33d2..ab7ca32 100644 --- a/skills/install.md +++ b/skills/install.md @@ -11,7 +11,7 @@ User-facing prompt: ## Goal Install the Knowject Skills pack for both Claude Code and Codex by linking all -`knowject-*` Skill directories into: +`knowject-*` Skill directories and the shared support folder `_shared` into: ```text ~/.claude/skills/ @@ -83,6 +83,9 @@ Check that the core Skill is available for both clients: ```bash test -f ~/.claude/skills/knowject-context-init/SKILL.md test -f ~/.codex/skills/knowject-context-init/SKILL.md +test -f ~/.claude/skills/_shared/schema.py +test -f ~/.codex/skills/_shared/schema.py +test -f ~/.codex/skills/knowject-context-init/../_shared/context-yaml-schema.md ``` If PyYAML is available, run the full Skill pack validation from the Knowject diff --git a/skills/knowject-mcp-tool-designer/README.md b/skills/knowject-mcp-tool-designer/README.md new file mode 100644 index 0000000..d69bccf --- /dev/null +++ b/skills/knowject-mcp-tool-designer/README.md @@ -0,0 +1,62 @@ +# knowject-mcp-tool-designer + +## What It Does + +`knowject-mcp-tool-designer` turns project API surfaces into MCP tool design artifacts. It maps endpoints into tool candidates with input schema, risk labels, confirmation gates, auth notes, audit notes, and rollback notes. + +Day 1 is design-only. The Skill writes reviewable files under `knowject/tools/`; it does not implement an MCP server, runtime handlers, platform API, UI, or auth code. + +## When To Use + +- You want to decide which API endpoints are safe to expose as agent tools. +- You need tool names, descriptions, input schemas, and source endpoint mappings. +- You want to separate read-only tools from write, destructive, sensitive, or external-side-effect tools. +- You need confirmation gate rules before implementing MCP handlers. + +## Inputs + +- `knowject/context.yaml`. +- OpenAPI or Express sources from `api.sources`. +- Endpoint inventory generated by `knowject-read-api`. +- Explicit user-provided endpoint lists. + +## Outputs + +- `knowject/tools/README.md`. +- `knowject/tools/mcp-tools.plan.md`. +- `knowject/tools/mcp-tools.schema.json`. +- `knowject/tools/tool-risk-report.md`. +- Unified diffs and confirmation prompts before writing. + +## Example + +User asks: + +```text +/knowject mcp tools from knowject-read-api inventory for users +``` + +Expected result: + +```text +The Skill maps selected endpoints into MCP tool candidates, labels read/write/destructive risk, adds confirmation gates where required, writes design artifacts under knowject/tools/ only after showing diffs, and reminds you that no MCP server was implemented. +``` + +## Safety Rules + +- Refuse without `knowject/context.yaml`, except for dry-run explanation. +- Never invent endpoints or call project/external APIs. +- Never copy secrets, tokens, DB URLs, auth headers, cookies, base URLs, or environment values. +- Require risk labels for every tool candidate. +- Require confirmation gates for destructive, external-side-effect, sensitive-data, and bulk operations. +- Use diff-confirm before writing. +- Day 1 does not implement MCP servers, runtime handlers, platform API, UI, auth logic, or permission code. + +## Related Files + +- [`SKILL.md`](./SKILL.md) +- [`agents/openai.yaml`](./agents/openai.yaml) +- [`references/tool-risk-taxonomy.md`](./references/tool-risk-taxonomy.md) +- [`references/mcp-tool-design-contract.md`](./references/mcp-tool-design-contract.md) +- [`references/api-selection.md`](./references/api-selection.md) +- [`references/examples/`](./references/examples/) diff --git a/skills/knowject-mcp-tool-designer/SKILL.md b/skills/knowject-mcp-tool-designer/SKILL.md new file mode 100644 index 0000000..9aa809c --- /dev/null +++ b/skills/knowject-mcp-tool-designer/SKILL.md @@ -0,0 +1,169 @@ +--- +name: knowject-mcp-tool-designer +description: | + Use when the user wants to design MCP tool candidates, function/tool calling + schemas, or agent tool risk gates from project API surfaces. Triggers on + phrases like "/knowject mcp tools", "design MCP tools from this API", "把这些 + endpoint 设计成 MCP tools", "为 agent 工具加确认 gate", "API-to-MCP tool design", + or "which endpoints are safe for tool calling". Day-1 output is design + artifacts only under knowject/tools/: README.md, mcp-tools.plan.md, + mcp-tools.schema.json, and tool-risk-report.md. It does not implement an MCP + server or runtime tool handlers. +--- + +# `knowject-mcp-tool-designer` + +You design MCP tool candidates from project API surfaces. The output is a conservative, reviewable set of design artifacts that map endpoints to tool names, input schemas, risk labels, confirmation gates, auth notes, audit notes, and rollback notes. + +## Hard rules + +1. **Refuse without `knowject/context.yaml`**, unless the user only asks for a dry-run explanation of the Skill. Point at `knowject-context-init` and stop. +2. **Design artifacts only in Day 1.** Do not implement an MCP server, runtime handlers, tool registration code, platform APIs, UI, auth logic, or permission checks. +3. **Never call external APIs or project endpoints.** Read API descriptions, inventories, and source files only. +4. **Never copy secrets or environment values.** Do not read or copy `.env*`, tokens, keys, passwords, cookies, auth headers, DB URLs, base URLs, private credentials, or raw sensitive records. +5. **Every tool candidate must cite its source endpoint.** Use a method/path plus source file, OpenAPI operation, endpoint inventory entry, or explicit user-provided endpoint. +6. **Risk labels are required.** Every candidate must use one of `read_only`, `low_write`, `destructive`, `external_side_effect`, or `sensitive_data`. +7. **Confirmation gates are required** for destructive operations, external side effects, sensitive data access, and bulk operations. If uncertain, require confirmation. +8. **Diff before write; confirm before write.** Show unified diffs for `knowject/tools/README.md`, `mcp-tools.plan.md`, `mcp-tools.schema.json`, and `tool-risk-report.md`; write only after explicit user confirmation. +9. **Use the user's language** for chat replies and Markdown summaries; JSON keys and enum values stay English. + +## Flow + +### Step 1 - Verify preconditions + +Confirm the current directory is the target repo root: + +```bash +git rev-parse --show-toplevel +``` + +Read `knowject/context.yaml`. + +- If it is missing and the user wants tool designs: refuse, point at `knowject-context-init`, and stop. +- If it is missing and the user asks only how the Skill works: explain the output contract and stop before reading API sources. +- Use `api.sources` and optional `api.client` when present. + +### Step 2 - Pick input mode + +Supported Day-1 input modes: + +| Mode | Use when | +|---|---| +| `context-openapi` | `knowject/context.yaml#api.sources[*].format` is `openapi`. | +| `context-express` | `api.sources[*].format` is `express`. | +| `endpoint-inventory` | The user provides output from `knowject-read-api`. | +| `explicit-endpoints` | The user provides a list of method/path endpoints. | + +If `api.sources` is missing or unsupported, ask for an endpoint inventory or explicit endpoint list. Do not guess endpoints. + +### Step 3 - Build or read endpoint inventory + +Apply [`references/api-selection.md`](./references/api-selection.md). + +Preferred source order: + +1. Existing endpoint inventory generated by `knowject-read-api`. +2. OpenAPI source from `context.yaml`. +3. Express route source from `context.yaml`. +4. Explicit user-provided endpoint list. + +When source format is supported by `knowject-read-api`, prefer its deterministic extractors and inventory shape. Do not change `knowject-read-api` behavior. + +### Step 4 - Select tool candidates + +For each candidate endpoint: + +- Prefer narrow, task-oriented read endpoints before write endpoints. +- Exclude endpoints whose behavior is unclear. +- Exclude auth/session, internal admin, raw import/export, and dangerous bulk endpoints unless the user explicitly asks to evaluate them. +- Keep read and write candidates separated in the plan. +- Attach a source endpoint citation before writing the candidate. + +### Step 5 - Assign schema, risk, and gates + +Use [`references/tool-risk-taxonomy.md`](./references/tool-risk-taxonomy.md). + +Each candidate must include: + +- `tool_name` +- `description` +- `source_endpoint` +- `input_schema` +- `output_shape_notes` +- `risk_level` +- `requires_confirmation` +- `auth_scope_notes` +- `audit_log_notes` +- `rollback_notes` + +Set `requires_confirmation: true` when: + +- the endpoint deletes, updates, sends, imports, exports, purchases, provisions, or triggers irreversible work +- the endpoint can affect external systems +- the endpoint can expose sensitive data +- the endpoint supports bulk operations +- the risk is uncertain + +### Step 6 - Prepare files + +Apply [`references/mcp-tool-design-contract.md`](./references/mcp-tool-design-contract.md). + +Write or update exactly these Day-1 files: + +```text +knowject/tools/ + README.md + mcp-tools.plan.md + mcp-tools.schema.json + tool-risk-report.md +``` + +### Step 7 - Diff and confirm + +Before writing: + +1. Validate `mcp-tools.schema.json` parses as JSON. +2. Show unified diffs for every create or update. +3. Ask: "Apply these MCP tool design artifacts? (yes / cancel)". +4. On `yes`, write the files. +5. On `cancel`, stop without writing anything. + +### Step 8 - Summarize + +After writing, report: + +- Tool candidates added or updated. +- Risk levels used. +- Confirmation gates required. +- Endpoints skipped, with reason. +- Any sensitive source intentionally not copied. +- Clear reminder that no MCP server or runtime handler was implemented. + +## Failure modes + +- **`knowject/context.yaml` missing:** refuse and tell the user to run `knowject-context-init` first. +- **No API sources or endpoints found:** ask for a `knowject-read-api` inventory or explicit endpoint list. +- **Unsupported source format:** explain supported Day-1 modes and ask for OpenAPI, Express, inventory, or explicit endpoints. +- **Endpoint behavior unclear:** skip it or mark it high-risk with confirmation required; do not invent behavior. +- **Sensitive or secret-bearing source:** skip sensitive values and warn the user. +- **Schema JSON invalid:** fix before showing the write confirmation. +- **User does not confirm the diff:** stop without writing. + +## What this Skill does NOT do + +- Does not implement an MCP server. +- Does not create runtime tool handlers or register tools in an agent runtime. +- Does not call external APIs or project endpoints. +- Does not modify app API routes, auth/permission code, platform UI, or backend services. +- Does not change `knowject-read-api` behavior. +- Does not claim that the designed tools are safe to run without implementation review, tests, auth checks, and audit integration. + +## See also + +- Risk taxonomy: [`references/tool-risk-taxonomy.md`](./references/tool-risk-taxonomy.md) +- File contract: [`references/mcp-tool-design-contract.md`](./references/mcp-tool-design-contract.md) +- API selection: [`references/api-selection.md`](./references/api-selection.md) +- Examples: [`references/examples/`](./references/examples/) +- API discovery: [`../knowject-read-api/SKILL.md`](../knowject-read-api/SKILL.md) +- API typing: [`../knowject-api-to-types/SKILL.md`](../knowject-api-to-types/SKILL.md) +- Schema reference: [`../_shared/context-yaml-schema.md`](../_shared/context-yaml-schema.md) diff --git a/skills/knowject-mcp-tool-designer/agents/openai.yaml b/skills/knowject-mcp-tool-designer/agents/openai.yaml new file mode 100644 index 0000000..3672516 --- /dev/null +++ b/skills/knowject-mcp-tool-designer/agents/openai.yaml @@ -0,0 +1,6 @@ +# skills/knowject-mcp-tool-designer/agents/openai.yaml +# Codex adapter for knowject-mcp-tool-designer Skill. +interface: + display_name: "Knowject MCP Tool Designer" + short_description: "Design API-to-MCP tool candidates with schemas, risk labels, confirmation gates, auth notes, and audit/rollback notes." + default_prompt: "Use $knowject-mcp-tool-designer to read knowject/context.yaml, inspect OpenAPI/Express sources, knowject-read-api endpoint inventories, or explicit endpoint lists, then propose knowject/tools/README.md, knowject/tools/mcp-tools.plan.md, knowject/tools/mcp-tools.schema.json, and knowject/tools/tool-risk-report.md. Include source_endpoint, input_schema, risk_level, requires_confirmation, auth_scope_notes, audit_log_notes, and rollback_notes for each tool candidate. Always show unified diffs and require explicit confirmation before writing. Keep Day-1 scope design-artifact only: no MCP server, runtime handlers, API calls, platform UI, auth code, or external API calls." diff --git a/skills/knowject-mcp-tool-designer/references/api-selection.md b/skills/knowject-mcp-tool-designer/references/api-selection.md new file mode 100644 index 0000000..32a2144 --- /dev/null +++ b/skills/knowject-mcp-tool-designer/references/api-selection.md @@ -0,0 +1,42 @@ +# API Selection For MCP Tool Design + +Start from the most structured API source available. Do not infer endpoints from UI labels or prose alone. + +## Supported Inputs + +| Input | How to use it | +|---|---| +| OpenAPI source from `knowject/context.yaml` | Read operations, methods, paths, operation ids, params, request bodies, and response summaries. | +| Express route source from `knowject/context.yaml` | Use `knowject-read-api` extractor output when possible; otherwise cite route files and lines. | +| Endpoint inventory from `knowject-read-api` | Preferred Day-1 input because it is already normalized. | +| Explicit endpoint list | Use only the endpoints the user provided; mark missing behavior as uncertain. | + +## Candidate Selection + +Prefer endpoints that: + +- have a clear single-purpose action +- have explicit path/query/body inputs +- can be described without environment-specific values +- have understandable auth scope +- have low blast radius + +Be cautious with endpoints that: + +- delete, archive, import, export, rebuild, retry, send, publish, or provision +- support broad filters or bulk mutation +- return sensitive or tenant-crossing data +- lack a clear rollback path +- depend on implicit server-side context not visible in the source + +## Naming + +Use stable snake_case names: + +- `GET /users` -> `list_users` +- `GET /users/{id}` -> `get_user` +- `POST /users` -> `create_user` +- `PATCH /users/{id}` -> `update_user` +- `DELETE /users/{id}` -> `delete_user` + +Prefer intent-revealing names over raw REST names when the endpoint is domain-specific, but keep the source endpoint explicit in the schema. diff --git a/skills/knowject-mcp-tool-designer/references/examples/mcp-tools.plan.expected.md b/skills/knowject-mcp-tool-designer/references/examples/mcp-tools.plan.expected.md new file mode 100644 index 0000000..6539f6a --- /dev/null +++ b/skills/knowject-mcp-tool-designer/references/examples/mcp-tools.plan.expected.md @@ -0,0 +1,43 @@ +# MCP Tool Design Plan + +Generated by `knowject-mcp-tool-designer` on 2026-05-24. + +## Source Summary + +- Source mode: `endpoint-inventory` +- Source: `skills/knowject-read-api/references/examples/express-expected.json` +- Day-1 output is design-only. No MCP server, runtime handler, auth logic, or API route change is included. + +## Candidate Tools + +| Tool | Endpoint | Risk | Confirmation | Notes | +|---|---|---|---|---| +| `list_users` | `GET /users` | `read_only` | no | Good first candidate if tenant scope is already enforced by the API. | +| `delete_user` | `DELETE /users/:id` | `destructive` | yes | Keep design-only until auth scope, audit log, and rollback behavior are implemented. | + +## Implementation Notes + +### `list_users` + +- Input schema has no required fields in the example inventory. +- Tool description must mention visible/current-project scope. +- Output shape should be filled from OpenAPI response schemas before implementation. + +### `delete_user` + +- Requires explicit confirmation naming the target user id. +- Requires privileged auth scope and tenant boundary checks. +- Requires audit logging before implementation. +- Requires a restore or re-create procedure before runtime exposure. + +## Skipped Endpoints + +- `POST /users` and `PATCH /users/:id` are write candidates but need body schema and rollback details before becoming tools. +- Multi-parameter order item endpoint is skipped in this example because it is unrelated to user management. + +## Day-1 Non-Goals + +- No MCP server implementation. +- No runtime tool handlers. +- No external API calls. +- No platform UI or backend route changes. diff --git a/skills/knowject-mcp-tool-designer/references/examples/mcp-tools.schema.expected.json b/skills/knowject-mcp-tool-designer/references/examples/mcp-tools.schema.expected.json new file mode 100644 index 0000000..381bf89 --- /dev/null +++ b/skills/knowject-mcp-tool-designer/references/examples/mcp-tools.schema.expected.json @@ -0,0 +1,64 @@ +{ + "version": "0.1", + "project": { + "name": "Knowject", + "generated_at": "2026-05-24", + "source_summary": { + "mode": "endpoint-inventory", + "refs": [ + { + "path": "skills/knowject-read-api/references/examples/express-expected.json", + "note": "example endpoint inventory generated by knowject-read-api" + } + ] + } + }, + "tools": [ + { + "tool_name": "list_users", + "description": "List users visible to the current project member.", + "source_endpoint": { + "method": "GET", + "path": "/users", + "source": "skills/knowject-read-api/references/examples/express-expected.json#GET /users" + }, + "input_schema": { + "type": "object", + "properties": {}, + "required": [] + }, + "output_shape_notes": "Returns the user list shape defined by the source API or OpenAPI response schema.", + "risk_level": "read_only", + "requires_confirmation": false, + "auth_scope_notes": "Requires project member read access; do not expose cross-tenant user lists.", + "audit_log_notes": "Standard request logging is sufficient for read-only inspection.", + "rollback_notes": "No rollback needed for read-only access." + }, + { + "tool_name": "delete_user", + "description": "Delete one user by id after explicit operator confirmation.", + "source_endpoint": { + "method": "DELETE", + "path": "/users/:id", + "source": "skills/knowject-read-api/references/examples/express-expected.json#DELETE /users/:id" + }, + "input_schema": { + "type": "object", + "properties": { + "id": { + "type": "string", + "description": "User id from the source endpoint path parameter." + } + }, + "required": ["id"], + "additionalProperties": false + }, + "output_shape_notes": "Returns a deletion acknowledgement or the deleted resource shape, depending on the API implementation.", + "risk_level": "destructive", + "requires_confirmation": true, + "auth_scope_notes": "Requires privileged user management scope and tenant boundary checks.", + "audit_log_notes": "Record actor, target user id, timestamp, request id, and confirmation text.", + "rollback_notes": "Define restore or re-create procedure before implementation; otherwise keep as design-only." + } + ] +} diff --git a/skills/knowject-mcp-tool-designer/references/examples/tool-risk-report.expected.md b/skills/knowject-mcp-tool-designer/references/examples/tool-risk-report.expected.md new file mode 100644 index 0000000..ba0b2e9 --- /dev/null +++ b/skills/knowject-mcp-tool-designer/references/examples/tool-risk-report.expected.md @@ -0,0 +1,31 @@ +# Tool Risk Report + +Generated by `knowject-mcp-tool-designer` on 2026-05-24. + +## Risk Summary + +| Risk level | Tools | +|---|---| +| `read_only` | `list_users` | +| `destructive` | `delete_user` | + +## Confirmation Gates + +- `list_users`: no confirmation in the example, assuming the API already scopes results to the current tenant or project. +- `delete_user`: confirmation required. Prompt must include tool name, target user id, source endpoint, and destructive risk reason. + +## Sensitive Data Concerns + +- User lists may expose personal or tenant-sensitive data. If the source API can return cross-tenant data, reclassify `list_users` as `sensitive_data`. +- Deletion must not expose credential, token, cookie, or auth header values in logs or artifacts. + +## Auth Scope Notes + +- `list_users` needs normal project member read access. +- `delete_user` needs privileged user management scope plus tenant boundary checks. + +## Audit And Rollback Notes + +- Read-only access can rely on standard request logs unless policy requires explicit audit events. +- Deletion needs actor id, target id, timestamp, request id, and confirmation text. +- Do not implement `delete_user` until restore or re-create behavior is documented. diff --git a/skills/knowject-mcp-tool-designer/references/mcp-tool-design-contract.md b/skills/knowject-mcp-tool-designer/references/mcp-tool-design-contract.md new file mode 100644 index 0000000..05c4356 --- /dev/null +++ b/skills/knowject-mcp-tool-designer/references/mcp-tool-design-contract.md @@ -0,0 +1,111 @@ +# MCP Tool Design Contract + +Day-1 output is design-artifact only and repo-local. + +```text +knowject/tools/ + README.md + mcp-tools.plan.md + mcp-tools.schema.json + tool-risk-report.md +``` + +## `mcp-tools.schema.json` + +Use this machine-readable shape: + +```json +{ + "version": "0.1", + "project": { + "name": "", + "generated_at": "YYYY-MM-DD", + "source_summary": { + "mode": "context-openapi | context-express | endpoint-inventory | explicit-endpoints", + "refs": [ + { + "path": "api-inventory/users.json", + "note": "knowject-read-api endpoint inventory" + } + ] + } + }, + "tools": [ + { + "tool_name": "list_users", + "description": "List users visible to the current project member.", + "source_endpoint": { + "method": "GET", + "path": "/users", + "source": "apps/api/src/modules/users/users.routes.ts:7" + }, + "input_schema": { + "type": "object", + "properties": {}, + "required": [] + }, + "output_shape_notes": "Returns a list shape defined by the source API.", + "risk_level": "read_only", + "requires_confirmation": false, + "auth_scope_notes": "Requires normal project member read scope.", + "audit_log_notes": "Standard request logging is sufficient.", + "rollback_notes": "No rollback needed for read-only tool." + } + ] +} +``` + +## Required Tool Fields + +Every tool candidate requires: + +- `tool_name` +- `description` +- `source_endpoint` +- `input_schema` +- `output_shape_notes` +- `risk_level` +- `requires_confirmation` +- `auth_scope_notes` +- `audit_log_notes` +- `rollback_notes` + +`source_endpoint` must include at least `method`, `path`, and `source`. + +## README Template + +Use this content when creating `knowject/tools/README.md`: + +```markdown +# knowject/tools/ + +This folder stores MCP tool design artifacts produced by `knowject-mcp-tool-designer`. + +- `mcp-tools.plan.md` explains the candidate tools and implementation notes. +- `mcp-tools.schema.json` is the machine-readable tool candidate file. +- `tool-risk-report.md` records risk labels, confirmation gates, auth notes, audit notes, and rollback notes. +- These files are design artifacts only. They do not implement an MCP server or runtime handlers. +- Do not store secrets, tokens, API keys, DB URLs, cookies, auth headers, base URLs, or environment-specific values here. + +Review diffs before committing tool design updates. +``` + +## Markdown Artifact Requirements + +`mcp-tools.plan.md` should include: + +- source summary +- candidate tool table +- per-tool implementation notes +- skipped endpoints +- explicit Day-1 non-goals + +`tool-risk-report.md` should include: + +- risk summary by level +- confirmation gates +- sensitive-data concerns +- auth scope notes +- audit and rollback notes + +Do not claim the tools are implemented or safe to run without implementation review. diff --git a/skills/knowject-mcp-tool-designer/references/tool-risk-taxonomy.md b/skills/knowject-mcp-tool-designer/references/tool-risk-taxonomy.md new file mode 100644 index 0000000..229697a --- /dev/null +++ b/skills/knowject-mcp-tool-designer/references/tool-risk-taxonomy.md @@ -0,0 +1,33 @@ +# MCP Tool Risk Taxonomy + +Use one risk label per tool candidate. When uncertain, choose the higher-risk label and require confirmation. + +## Risk Levels + +| Risk level | Meaning | Confirmation | +|---|---|---| +| `read_only` | Reads non-sensitive data and has no side effects. | Usually not required, unless data sensitivity is unclear. | +| `low_write` | Creates or updates low-impact project data with a clear undo path. | Required when user-visible state changes. | +| `destructive` | Deletes, archives, resets, overwrites, bulk updates, or triggers irreversible work. | Always required. | +| `external_side_effect` | Sends messages, provisions resources, calls third-party systems, charges money, or notifies users. | Always required. | +| `sensitive_data` | Reads or writes private, credential-like, regulated, tenant-sensitive, or personally sensitive data. | Always required. | + +## Confirmation Gates + +Require confirmation for: + +- destructive operations +- external side effects +- sensitive data access +- bulk operations +- tenant-crossing operations +- unclear endpoint behavior + +The confirmation prompt should name the tool, affected resource, source endpoint, and risk reason. + +## Selection Guidance + +- Prefer `read_only` tools for Day-1 implementation candidates. +- Keep write tools in the design plan only until auth, audit logging, tests, and rollback behavior are clear. +- Do not design tools for auth/session endpoints unless the user explicitly asks and the risk report labels them high risk. +- Never hide uncertainty by using a lower risk label. diff --git a/skills/knowject-rag-eval/README.md b/skills/knowject-rag-eval/README.md new file mode 100644 index 0000000..cabea6f --- /dev/null +++ b/skills/knowject-rag-eval/README.md @@ -0,0 +1,61 @@ +# knowject-rag-eval + +## What It Does + +`knowject-rag-eval` creates lightweight, source-cited RAG and citation evaluation artifacts from project knowledge sources. It helps teams review whether retrieval should find the right documents, whether answers should cite evidence, and where unsupported claims or stale roadmap facts may appear. + +Day 1 is file-based: the Skill writes reviewable eval files under `knowject/evals/` and does not run a live chatbot or scoring engine. + +## When To Use + +- You want a small eval set before changing prompts, chunking, retrieval, or citation behavior. +- You want to test source recall, citation support, unsupported-claim avoidance, or fact-vs-plan conflicts. +- You want to turn current docs, contracts, standards, or project memory into reusable RAG review cases. +- You need a safe review artifact without adding a database, vector store, or external eval dependency. + +## Inputs + +- `knowject/context.yaml`. +- Stable knowledge sources such as `docs/current/**`, `docs/contracts/**`, and `docs/standards/**`. +- Optional `knowject/memory/project-memory.yaml`. +- User-provided knowledge files or requested git ranges. + +## Outputs + +- `knowject/evals/README.md`. +- `knowject/evals/rag-eval-cases.yaml`. +- `knowject/evals/rag-eval-report.md`. +- Unified diffs and confirmation prompts before writing. + +## Example + +User asks: + +```text +/knowject rag eval from docs/current/architecture.md docs/contracts/chat-contract.md +``` + +Expected result: + +```text +The Skill creates source-cited eval cases for retrieval, citation support, unsupported claims, and fact-vs-plan risks, validates rag-eval-cases.yaml, shows diffs, and writes knowject/evals/ only after confirmation. +``` + +## Safety Rules + +- Refuse without `knowject/context.yaml`, except for dry-run explanation. +- Require source evidence for every eval case. +- Never copy secrets, tokens, DB URLs, auth headers, cookies, base URLs, or environment values. +- Prefer current docs and contracts over roadmap or old handoff notes. +- Use diff-confirm before writing. +- Day 1 does not call models, query Chroma/MongoDB, add UI/API code, or run external scoring. + +## Related Files + +- [`SKILL.md`](./SKILL.md) +- [`agents/openai.yaml`](./agents/openai.yaml) +- [`references/eval-taxonomy.md`](./references/eval-taxonomy.md) +- [`references/eval-file-contract.md`](./references/eval-file-contract.md) +- [`references/source-selection.md`](./references/source-selection.md) +- [`scripts/validate-rag-eval-cases.py`](./scripts/validate-rag-eval-cases.py) +- [`references/examples/`](./references/examples/) diff --git a/skills/knowject-rag-eval/SKILL.md b/skills/knowject-rag-eval/SKILL.md new file mode 100644 index 0000000..9d61b90 --- /dev/null +++ b/skills/knowject-rag-eval/SKILL.md @@ -0,0 +1,182 @@ +--- +name: knowject-rag-eval +description: | + Use when the user wants to create or review lightweight RAG, retrieval, and + citation-quality evaluation cases from project knowledge artifacts. Triggers + on phrases like "/knowject rag eval", "evaluate RAG citations", "生成 RAG + 评测用例", "检查引用是否有来源", "把这些知识库文档做成 eval cases", or "评估事实和 + roadmap 是否混淆". Day-1 output is file-based only under knowject/evals/: + README.md, rag-eval-cases.yaml, and rag-eval-report.md. Every eval case must + include source evidence, avoid secrets, and pass a diff-confirm gate before + writing. +--- + +# `knowject-rag-eval` + +You generate reviewable RAG and citation evaluation artifacts from selected project knowledge sources. The output is a small file-based eval set under `knowject/evals/` that helps teams test retrieval, citation support, unsupported-claim avoidance, and fact-vs-plan conflicts before changing prompts, chunking, or retrieval behavior. + +## Hard rules + +1. **Refuse without `knowject/context.yaml`**, unless the user only asks for a dry-run explanation of the Skill. Point at `knowject-context-init` and stop. +2. **Every eval case needs source evidence.** Each case must include `expected_source_refs` with repo-relative paths plus line ranges or notes. Do not write source-free questions. +3. **No live scoring in Day 1.** Do not call LLMs, chatbots, Chroma, MongoDB, platform APIs, or external evaluation services. +4. **Never copy secrets or environment values.** Do not read or copy `.env*`, tokens, keys, passwords, cookies, auth headers, DB URLs, base URLs, private credentials, or raw sensitive records. +5. **Current facts override roadmap, plans, and old handoffs.** Eval cases that test conflict resolution must encode this precedence explicitly. +6. **Diff before write; confirm before write.** Show unified diffs for `knowject/evals/README.md`, `rag-eval-cases.yaml`, and `rag-eval-report.md`; write only after explicit user confirmation. +7. **Do not scan outside the current repo** unless the user provides exact paths. Never scan the whole home directory. +8. **Use the user's language** for chat replies and question wording when possible; YAML keys and enum values stay English. +9. **Keep the scope file-based and reviewable.** This Skill generates eval artifacts; it does not prove live RAG quality by itself. + +## Flow + +### Step 1 - Verify preconditions + +Confirm the current directory is the target repo root: + +```bash +git rev-parse --show-toplevel +``` + +Read `knowject/context.yaml`. + +- If it is missing and the user wants eval artifacts: refuse, point at `knowject-context-init`, and stop. +- If it is missing and the user asks only how the Skill works: explain the contract and stop before reading sources. +- If it exists: use `project.name` when preparing `rag-eval-cases.yaml`. + +### Step 2 - Pick source mode + +Choose one mode from the request: + +| Mode | Use when | +|---|---| +| `explicit-files` | The user names knowledge docs, memory files, contracts, or directories. | +| `default-scan` | The user asks for RAG evals without naming sources. | +| `memory-assisted` | The user asks to build evals from `knowject/memory/project-memory.yaml`. | +| `git-range` | The user asks to make evals from a branch, commit, diff, or PR. | + +If the source set is too broad, ask one narrowing question. Prefer fewer high-signal evals over broad, shallow coverage. + +### Step 3 - Select sources + +Apply [`references/source-selection.md`](./references/source-selection.md). + +Stable source priority: + +1. `knowject/context.yaml` +2. `docs/current/**` +3. `docs/contracts/**` +4. `docs/standards/**` +5. `knowject/memory/project-memory.yaml` +6. User-provided knowledge files +7. `docs/handoff/**` and roadmap docs only when testing stale-source or fact-vs-plan behavior + +Avoid `.env*`, logs, generated build output, `.git/`, `node_modules/`, shell history, browser history, private chat logs, and any path likely to contain secrets. + +### Step 4 - Draft eval cases + +Use [`references/eval-taxonomy.md`](./references/eval-taxonomy.md) and create cases with these Day-1 eval types: + +- `source_recall` +- `citation_support` +- `unsupported_claim` +- `fact_vs_plan` +- `conflict_resolution` + +For each case: + +- Ask one focused question that a RAG answer should handle. +- Attach the expected source refs before writing expected answer points. +- List the answer points the cited sources support. +- List forbidden claims that a model must not invent. +- Set `difficulty` to `easy`, `medium`, or `hard`. +- Add compact tags such as `architecture`, `contract`, `memory`, `citation`, or `roadmap`. + +### Step 5 - Prepare the report + +Create `rag-eval-report.md` with: + +- coverage by source +- source quality risks +- citation risk +- chunking / retrieval recommendations +- unsupported-claim watchlist + +The report is a review artifact, not a numeric benchmark. Do not claim the project has passed live RAG evaluation unless the user later runs these cases against a system. + +### Step 6 - Prepare files + +Apply [`references/eval-file-contract.md`](./references/eval-file-contract.md). + +Write or update exactly these Day-1 files: + +```text +knowject/evals/ + README.md + rag-eval-cases.yaml + rag-eval-report.md +``` + +Use ids like: + +```text +rag-YYYYMMDD-001 +``` + +Keep existing ids stable when updating an existing eval set. + +### Step 7 - Validate, diff, and confirm + +Before writing: + +1. Validate proposed cases: + + ```bash + python3 /knowject-rag-eval/scripts/validate-rag-eval-cases.py \ + + ``` + +2. Show unified diffs for all proposed file creates or updates. +3. Ask: "Apply these RAG eval changes? (yes / cancel)". +4. On `yes`, write the files. +5. On `cancel`, stop without writing anything. + +If validation fails, show the errors and fix the proposal before asking for confirmation. + +### Step 8 - Summarize + +After writing, report: + +- Cases added or updated. +- Sources covered. +- Sources skipped, with reason. +- Any sensitive source intentionally not copied. +- Suggested next step, such as running the cases manually against project chat or adding them to review docs. + +## Failure modes + +- **`knowject/context.yaml` missing:** refuse and tell the user to run `knowject-context-init` first. +- **No durable sources found:** explain checked paths and ask for explicit knowledge files. +- **Source set is too broad:** ask one narrowing question instead of scanning everything. +- **Source appears secret-bearing:** skip secret values and warn the user. +- **Case has no evidence:** skip it or ask for a source. Do not write it. +- **Current and old sources conflict:** make a `fact_vs_plan` or `conflict_resolution` case, with current docs/contracts as the expected source. +- **Existing eval YAML is invalid:** stop, show validation errors, and ask whether to repair it before adding cases. +- **User does not confirm the diff:** stop without writing. + +## What this Skill does NOT do + +- Does not run live model calls or automated chatbot scoring. +- Does not query Chroma, MongoDB, vector stores, APIs, or platform runtime. +- Does not add platform UI, backend routes, CI requiring network, or external eval dependencies such as RAGAS. +- Does not claim production RAG quality, production scoring, or complete evaluation coverage. +- Does not copy secrets or environment-specific values into eval files. + +## See also + +- Eval taxonomy: [`references/eval-taxonomy.md`](./references/eval-taxonomy.md) +- File contract: [`references/eval-file-contract.md`](./references/eval-file-contract.md) +- Source selection: [`references/source-selection.md`](./references/source-selection.md) +- Case validator: [`scripts/validate-rag-eval-cases.py`](./scripts/validate-rag-eval-cases.py) +- Examples: [`references/examples/`](./references/examples/) +- Memory capture: [`../knowject-memory-capture/SKILL.md`](../knowject-memory-capture/SKILL.md) +- Schema reference: [`../_shared/context-yaml-schema.md`](../_shared/context-yaml-schema.md) diff --git a/skills/knowject-rag-eval/agents/openai.yaml b/skills/knowject-rag-eval/agents/openai.yaml new file mode 100644 index 0000000..66738af --- /dev/null +++ b/skills/knowject-rag-eval/agents/openai.yaml @@ -0,0 +1,6 @@ +# skills/knowject-rag-eval/agents/openai.yaml +# Codex adapter for knowject-rag-eval Skill. +interface: + display_name: "Knowject RAG Eval" + short_description: "Create source-cited RAG and citation evaluation cases from project knowledge artifacts under knowject/evals/." + default_prompt: "Use $knowject-rag-eval to read knowject/context.yaml, select stable project knowledge sources, generate source-cited RAG/citation eval cases, and propose knowject/evals/README.md, knowject/evals/rag-eval-cases.yaml, and knowject/evals/rag-eval-report.md. Validate proposed rag-eval-cases.yaml with scripts/validate-rag-eval-cases.py. Always show unified diffs and require explicit confirmation before writing. Keep Day-1 scope file-based only: no live model calls, Chroma, MongoDB, API, UI, network scoring, or external eval dependencies." diff --git a/skills/knowject-rag-eval/references/eval-file-contract.md b/skills/knowject-rag-eval/references/eval-file-contract.md new file mode 100644 index 0000000..2846cb4 --- /dev/null +++ b/skills/knowject-rag-eval/references/eval-file-contract.md @@ -0,0 +1,101 @@ +# RAG Eval File Contract + +Day-1 RAG eval output is file-based and repo-local. + +```text +knowject/evals/ + README.md + rag-eval-cases.yaml + rag-eval-report.md +``` + +`rag-eval-cases.yaml` is the canonical machine-readable artifact. `rag-eval-report.md` is a human review report. + +## `rag-eval-cases.yaml` + +Use this shape: + +```yaml +version: "0.1" +project: + name: "" + generated_at: "YYYY-MM-DD" + source_summary: + mode: "default-scan | explicit-files | memory-assisted | git-range" + refs: + - path: "docs/current/architecture.md" + note: "current architecture facts" + +cases: + - id: "rag-YYYYMMDD-001" + eval_type: "source_recall | citation_support | unsupported_claim | fact_vs_plan | conflict_resolution" + difficulty: "easy | medium | hard" + question: "What should the answer prove?" + expected_source_refs: + - path: "docs/current/architecture.md" + line_start: 10 + line_end: 24 + expected_answer_points: + - "Supported point that should appear in the answer." + forbidden_claims: + - "Unsupported detail that must not appear." + tags: + - "architecture" + - "citation" +``` + +## Required Fields + +Every case requires: + +- `id` +- `eval_type` +- `difficulty` +- `question` +- `expected_source_refs` +- `expected_answer_points` +- `forbidden_claims` +- `tags` + +Every `expected_source_refs` entry requires `path` and either: + +- `line_start` + `line_end`, or +- `note` + +## README Template + +Use this content when creating `knowject/evals/README.md`: + +```markdown +# knowject/evals/ + +This folder stores source-cited RAG and citation evaluation artifacts produced by `knowject-rag-eval`. + +- `rag-eval-cases.yaml` is the canonical machine-readable eval case file. +- `rag-eval-report.md` summarizes source coverage, citation risks, chunking/retrieval recommendations, and unsupported-claim watchlists. +- Every eval case must include source evidence. +- This folder must not contain secrets, tokens, API keys, DB URLs, cookies, auth headers, or environment-specific values. + +Review diffs before committing eval updates. +``` + +## Report Template + +`rag-eval-report.md` should include: + +- source coverage +- source quality risks +- citation risks +- chunking / retrieval recommendations +- unsupported-claim watchlist + +Keep the report factual. Do not claim live RAG scoring or model performance. + +## Validation + +Validate generated cases before writing or committing: + +```bash +python3 /knowject-rag-eval/scripts/validate-rag-eval-cases.py \ + knowject/evals/rag-eval-cases.yaml +``` diff --git a/skills/knowject-rag-eval/references/eval-taxonomy.md b/skills/knowject-rag-eval/references/eval-taxonomy.md new file mode 100644 index 0000000..f56c0cd --- /dev/null +++ b/skills/knowject-rag-eval/references/eval-taxonomy.md @@ -0,0 +1,29 @@ +# RAG Eval Taxonomy + +Day-1 eval cases are designed for human review and later manual execution against a RAG or project-chat system. They are not live automated scoring. + +## Eval Types + +| Type | Purpose | Good case shape | +|---|---|---| +| `source_recall` | Check that retrieval should find the expected docs or chunks. | Question points to a specific documented fact and lists the expected source refs. | +| `citation_support` | Check that answer claims are supported by cited sources. | Expected answer points map directly to source refs. | +| `unsupported_claim` | Check that the model avoids details missing from sources. | Forbidden claims list common guesses or tempting but unsupported details. | +| `fact_vs_plan` | Check that current facts are not confused with roadmap or old handoff notes. | Current source refs and stale/plan refs are both named when useful. | +| `conflict_resolution` | Check that stronger sources override weaker or older sources. | Expected answer points state the precedence rule. | + +## Difficulty + +Use: + +- `easy` when one current source answers the question directly. +- `medium` when the answer needs two sources or careful citation boundaries. +- `hard` when the case involves conflicting docs, old plans, or unsupported-claim traps. + +## Case Quality Rules + +- One case should test one behavior. +- Prefer concrete questions over broad prompts. +- Keep expected answer points short and citeable. +- Put hallucination traps in `forbidden_claims`, not in the expected answer. +- Do not create cases from secrets, environment config, logs, or private credentials. diff --git a/skills/knowject-rag-eval/references/examples/rag-eval-cases.expected.yaml b/skills/knowject-rag-eval/references/examples/rag-eval-cases.expected.yaml new file mode 100644 index 0000000..4078cda --- /dev/null +++ b/skills/knowject-rag-eval/references/examples/rag-eval-cases.expected.yaml @@ -0,0 +1,48 @@ +version: "0.1" +project: + name: Knowject + generated_at: "2026-05-24" + source_summary: + mode: explicit-files + refs: + - path: docs/current/architecture.md + note: current architecture facts + - path: docs/roadmap/agent-workspace.md + note: roadmap facts used only for fact-vs-plan contrast +cases: + - id: rag-20260524-001 + eval_type: source_recall + difficulty: easy + question: Which service owns project knowledge indexing in Knowject? + expected_source_refs: + - path: docs/current/architecture.md + line_start: 42 + line_end: 58 + expected_answer_points: + - Project knowledge indexing is handled by the Python indexing runtime, not by the platform UI. + - The answer should cite the current architecture source. + forbidden_claims: + - The frontend directly writes embeddings. + - A roadmap document is enough to prove current indexing ownership. + tags: + - architecture + - source_recall + - id: rag-20260524-002 + eval_type: fact_vs_plan + difficulty: hard + question: Is Knowject already a fully implemented multi-step agent orchestration system? + expected_source_refs: + - path: docs/current/architecture.md + note: current implementation facts + - path: docs/roadmap/agent-workspace.md + note: roadmap plans that must not override current facts + expected_answer_points: + - The answer should distinguish shipped current capabilities from roadmap plans. + - The answer should avoid calling roadmap ideas current implementation. + forbidden_claims: + - Knowject has fully shipped multi-tool orchestration today. + - Roadmap-only multi-tool orchestration is already shipped. + tags: + - fact_vs_plan + - roadmap + - unsupported_claim diff --git a/skills/knowject-rag-eval/references/examples/rag-eval-report.expected.md b/skills/knowject-rag-eval/references/examples/rag-eval-report.expected.md new file mode 100644 index 0000000..5486eea --- /dev/null +++ b/skills/knowject-rag-eval/references/examples/rag-eval-report.expected.md @@ -0,0 +1,28 @@ +# RAG Eval Report + +Generated by `knowject-rag-eval` on 2026-05-24. + +## Source Coverage + +- `docs/current/architecture.md`: current architecture facts and implementation boundaries. +- `docs/roadmap/agent-workspace.md`: roadmap-only plans used for fact-vs-plan contrast. + +## Source Quality Risks + +- Roadmap and handoff docs can be useful negative controls, but they must not override current docs or contracts. +- If line ranges are unavailable, the eval case should include a specific note explaining why the source supports the expected answer. + +## Citation Risks + +- Answers may cite broad architecture docs while adding implementation details that are not present in the cited range. +- Answers may collapse roadmap plans into current implementation claims. + +## Chunking And Retrieval Recommendations + +- Keep current architecture chunks separate from roadmap chunks when possible. +- Preserve headings and file paths in chunk metadata so answers can cite the stronger source. + +## Unsupported-Claim Watchlist + +- Do not claim Knowject has fully shipped multi-tool orchestration unless a current implementation source proves it. +- Do not infer runtime ownership from UI copy or roadmap text alone. diff --git a/skills/knowject-rag-eval/references/source-selection.md b/skills/knowject-rag-eval/references/source-selection.md new file mode 100644 index 0000000..c749c1d --- /dev/null +++ b/skills/knowject-rag-eval/references/source-selection.md @@ -0,0 +1,36 @@ +# RAG Eval Source Selection + +Use the smallest source set that can produce useful, citeable eval cases. + +## Preferred Sources + +Read these first when present and relevant: + +1. `knowject/context.yaml` +2. `docs/current/**` +3. `docs/contracts/**` +4. `docs/standards/**` +5. `knowject/memory/project-memory.yaml` +6. user-provided knowledge files + +Use `docs/handoff/**`, `docs/roadmap/**`, or plans only when the eval case is intentionally testing stale-source handling, `fact_vs_plan`, or `conflict_resolution`. + +## Avoid + +Do not read or copy: + +- `.env*` +- credentials, tokens, cookies, auth headers, DB URLs, base URLs, or private keys +- `.git/` +- `node_modules/` +- build outputs +- logs that may contain sensitive values +- shell history, browser history, or private chat logs +- whole home directories or broad parent directories + +## Evidence Rules + +- Prefer repo-relative paths and line ranges. +- If a line range is not practical, include a short `note`. +- Do not create a case if the expected answer cannot be tied to source evidence. +- For conflict cases, cite both sides when possible and state which source should win. diff --git a/skills/knowject-rag-eval/scripts/validate-rag-eval-cases.py b/skills/knowject-rag-eval/scripts/validate-rag-eval-cases.py new file mode 100755 index 0000000..84b14e7 --- /dev/null +++ b/skills/knowject-rag-eval/scripts/validate-rag-eval-cases.py @@ -0,0 +1,181 @@ +#!/usr/bin/env python3 +"""Validate knowject/evals/rag-eval-cases.yaml.""" + +from __future__ import annotations + +import re +import sys +from pathlib import Path +from typing import Any + +try: + import yaml # type: ignore +except ImportError: + print("PyYAML is required. Install it: pip install pyyaml", file=sys.stderr) + sys.exit(3) + +EVAL_TYPES = { + "source_recall", + "citation_support", + "unsupported_claim", + "fact_vs_plan", + "conflict_resolution", +} +DIFFICULTY = {"easy", "medium", "hard"} +MODES = {"default-scan", "explicit-files", "memory-assisted", "git-range"} +DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$") +FORBIDDEN_KEY_RE = re.compile( + r"(api[_-]?key|auth[_-]?header|base[_-]?url|cookie|credential|database[_-]?url|db[_-]?url|password|secret|token)", + re.IGNORECASE, +) + + +def is_str(value: Any) -> bool: + return isinstance(value, str) and bool(value.strip()) + + +def add(errors: list[str], path: str, msg: str) -> None: + errors.append(f"{path}: {msg}") + + +def check_forbidden_keys(value: Any, path: str, errors: list[str]) -> None: + if isinstance(value, dict): + for key, child in value.items(): + key_path = f"{path}.{key}" if path else str(key) + if isinstance(key, str) and FORBIDDEN_KEY_RE.search(key): + add(errors, key_path, "forbidden secret/environment key") + check_forbidden_keys(child, key_path, errors) + elif isinstance(value, list): + for idx, child in enumerate(value): + check_forbidden_keys(child, f"{path}[{idx}]", errors) + + +def require_date(value: Any, path: str, errors: list[str]) -> None: + if not (isinstance(value, str) and DATE_RE.match(value)): + add(errors, path, "expected YYYY-MM-DD") + + +def validate_source_ref(ref: Any, path: str, errors: list[str]) -> None: + if not isinstance(ref, dict): + add(errors, path, "expected mapping") + return + if not is_str(ref.get("path")): + add(errors, f"{path}.path", "required non-empty string") + + has_note = is_str(ref.get("note")) + has_lines = "line_start" in ref and "line_end" in ref + if has_lines: + start = ref.get("line_start") + end = ref.get("line_end") + if not isinstance(start, int) or start <= 0: + add(errors, f"{path}.line_start", "expected positive integer") + if not isinstance(end, int) or end <= 0: + add(errors, f"{path}.line_end", "expected positive integer") + if isinstance(start, int) and isinstance(end, int) and end < start: + add(errors, path, "line_end must be >= line_start") + if not has_lines and not has_note: + add(errors, path, "requires line_start+line_end or note") + + +def require_string_list(value: Any, path: str, errors: list[str], *, allow_empty: bool = False) -> None: + if not isinstance(value, list): + add(errors, path, "expected list[str]") + return + if not allow_empty and not value: + add(errors, path, "required non-empty list") + return + for idx, item in enumerate(value): + if not is_str(item): + add(errors, f"{path}[{idx}]", "expected non-empty string") + + +def validate_case(case: Any, idx: int, errors: list[str]) -> None: + path = f"cases[{idx}]" + if not isinstance(case, dict): + add(errors, path, "expected mapping") + return + for key in ("id", "question"): + if not is_str(case.get(key)): + add(errors, f"{path}.{key}", "required non-empty string") + if case.get("eval_type") not in EVAL_TYPES: + add(errors, f"{path}.eval_type", f"must be one of {sorted(EVAL_TYPES)}") + if case.get("difficulty") not in DIFFICULTY: + add(errors, f"{path}.difficulty", f"must be one of {sorted(DIFFICULTY)}") + + refs = case.get("expected_source_refs") + if not isinstance(refs, list) or not refs: + add(errors, f"{path}.expected_source_refs", "required non-empty list") + else: + for ridx, ref in enumerate(refs): + validate_source_ref(ref, f"{path}.expected_source_refs[{ridx}]", errors) + + require_string_list(case.get("expected_answer_points"), f"{path}.expected_answer_points", errors) + require_string_list(case.get("forbidden_claims"), f"{path}.forbidden_claims", errors, allow_empty=True) + require_string_list(case.get("tags"), f"{path}.tags", errors) + + +def validate(doc: Any) -> list[str]: + errors: list[str] = [] + if not isinstance(doc, dict): + return ["root: expected mapping"] + + check_forbidden_keys(doc, "", errors) + + if doc.get("version") != "0.1": + add(errors, "version", "must be '0.1'") + + project = doc.get("project") + if not isinstance(project, dict): + add(errors, "project", "expected mapping") + else: + if not is_str(project.get("name")): + add(errors, "project.name", "required non-empty string") + require_date(project.get("generated_at"), "project.generated_at", errors) + summary = project.get("source_summary") + if not isinstance(summary, dict): + add(errors, "project.source_summary", "expected mapping") + else: + if summary.get("mode") not in MODES: + add(errors, "project.source_summary.mode", f"must be one of {sorted(MODES)}") + refs = summary.get("refs") + if not isinstance(refs, list) or not refs: + add(errors, "project.source_summary.refs", "required non-empty list") + else: + for idx, ref in enumerate(refs): + validate_source_ref(ref, f"project.source_summary.refs[{idx}]", errors) + + cases = doc.get("cases") + if not isinstance(cases, list) or not cases: + add(errors, "cases", "required non-empty list") + else: + for idx, case in enumerate(cases): + validate_case(case, idx, errors) + + return errors + + +def main() -> int: + if len(sys.argv) != 2: + print("usage: validate-rag-eval-cases.py ", file=sys.stderr) + return 2 + path = Path(sys.argv[1]) + if not path.is_file(): + print(f"not a file: {path}", file=sys.stderr) + return 2 + try: + doc = yaml.safe_load(path.read_text()) + except yaml.YAMLError as exc: + print(f"YAML parse failed: {exc}", file=sys.stderr) + return 2 + errors = validate(doc) + if errors: + print(f"INVALID: {path}", file=sys.stderr) + for error in errors: + print(f" - {error}", file=sys.stderr) + return 1 + print(f"OK: {path}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/skills/manifest.yaml b/skills/manifest.yaml index acd359f..7ae4599 100644 --- a/skills/manifest.yaml +++ b/skills/manifest.yaml @@ -15,6 +15,8 @@ skills: - knowject-read-design - knowject-api-to-types - knowject-memory-capture + - knowject-rag-eval + - knowject-mcp-tool-designer # Roadmap entries land here when promoted to Active in ROADMAP.md. # See ROADMAP.md for the Active / Deferred catalog and the decision log. roadmap: [] diff --git a/skills/scripts/install.sh b/skills/scripts/install.sh index d400f1b..c8a9b44 100755 --- a/skills/scripts/install.sh +++ b/skills/scripts/install.sh @@ -1,6 +1,7 @@ #!/usr/bin/env bash # skills/scripts/install.sh -# Symlink each knowject-* Skill into ~/.claude/skills/ and ~/.codex/skills/. +# Symlink each knowject-* Skill and shared support files into +# ~/.claude/skills/ and ~/.codex/skills/. # Idempotent: re-running replaces existing symlinks. # Safe: refuses to overwrite non-symlink files. @@ -17,27 +18,36 @@ mkdir -p "$CLAUDE_SKILLS" "$CODEX_SKILLS" installed=0 skipped=0 +link_entry() { + local source="$1" + local target="$2" + + if [ -L "$target" ]; then + rm "$target" + elif [ -e "$target" ]; then + echo "⚠️ Refuse to overwrite non-symlink: $target" >&2 + skipped=$((skipped + 1)) + return 0 + fi + + ln -s "$source" "$target" + installed=$((installed + 1)) + echo "→ linked $target → $source" +} + for skill_dir in "$SKILLS_DIR"/knowject-*/; do [ -d "$skill_dir" ] || continue skill_name="$(basename "$skill_dir")" for target_root in "$CLAUDE_SKILLS" "$CODEX_SKILLS"; do - target="$target_root/$skill_name" - - if [ -L "$target" ]; then - rm "$target" - elif [ -e "$target" ]; then - echo "⚠️ Refuse to overwrite non-symlink: $target" >&2 - skipped=$((skipped + 1)) - continue - fi - - ln -s "$skill_dir" "$target" - installed=$((installed + 1)) - echo "→ linked $target → $skill_dir" + link_entry "$skill_dir" "$target_root/$skill_name" done done +for target_root in "$CLAUDE_SKILLS" "$CODEX_SKILLS"; do + link_entry "$SKILLS_DIR/_shared" "$target_root/_shared" +done + echo "" echo "Installed: $installed symlinks" echo "Skipped: $skipped (non-symlink files in target — remove manually then re-run)" diff --git a/skills/scripts/test-install.sh b/skills/scripts/test-install.sh index 23d20bc..0be8612 100755 --- a/skills/scripts/test-install.sh +++ b/skills/scripts/test-install.sh @@ -15,6 +15,9 @@ HOME="$TEST_HOME" bash "$SCRIPT_DIR/install.sh" > /dev/null # Assert symlinks exist for tool in .claude .codex; do link="$TEST_HOME/$tool/skills/knowject-context-init" + shared_link="$TEST_HOME/$tool/skills/_shared" + skill_relative_schema="$link/../_shared/context-yaml-schema.md" + if [ ! -L "$link" ]; then echo "FAIL: $link is not a symlink" exit 1 @@ -27,8 +30,34 @@ for tool in .claude .codex; do echo "FAIL: $link/SKILL.md missing through symlink" exit 1 fi + if [ ! -L "$shared_link" ]; then + echo "FAIL: $shared_link is not a symlink" + exit 1 + fi + if [ ! -f "$shared_link/schema.py" ]; then + echo "FAIL: $shared_link/schema.py missing through symlink" + exit 1 + fi + if [ ! -f "$skill_relative_schema" ]; then + echo "FAIL: $skill_relative_schema does not resolve" + exit 1 + fi done # Test idempotency: run again, should not fail HOME="$TEST_HOME" bash "$SCRIPT_DIR/install.sh" > /dev/null + +# Test safety: non-symlink targets must not be overwritten. +BLOCK_HOME="$TEST_HOME/block" +mkdir -p "$BLOCK_HOME/.claude/skills" "$BLOCK_HOME/.codex/skills" +mkdir "$BLOCK_HOME/.codex/skills/_shared" +if HOME="$BLOCK_HOME" bash "$SCRIPT_DIR/install.sh" > /dev/null 2>&1; then + echo "FAIL: install.sh overwrote or ignored a blocking non-symlink target" + exit 1 +fi +if [ ! -d "$BLOCK_HOME/.codex/skills/_shared" ] || [ -L "$BLOCK_HOME/.codex/skills/_shared" ]; then + echo "FAIL: blocking non-symlink _shared target was overwritten" + exit 1 +fi + echo "OK: install.sh is correct and idempotent" diff --git a/skills/scripts/verify.sh b/skills/scripts/verify.sh index 3a07b86..9a3685c 100755 --- a/skills/scripts/verify.sh +++ b/skills/scripts/verify.sh @@ -426,6 +426,94 @@ else fail "uncited memory did not report expected source_refs error" fi +# 20. validate-rag-eval-cases.py accepts the rag eval example +note "validate-rag-eval-cases.py accepts rag eval example" +rag_eval_example="$SKILLS_DIR/knowject-rag-eval/references/examples/rag-eval-cases.expected.yaml" +if [ -f "$rag_eval_example" ]; then + if python3 "$SKILLS_DIR/knowject-rag-eval/scripts/validate-rag-eval-cases.py" "$rag_eval_example" >/dev/null 2>&1; then + pass "rag-eval example validates" + else + fail "rag-eval example YAML invalid" + fi +else + fail "rag-eval example missing" +fi + +# 21. validate-rag-eval-cases.py rejects uncited eval cases +note "validate-rag-eval-cases.py rejects uncited eval cases" +bad_rag_eval="$TMP_DIR/bad-rag-eval-cases.yaml" +cat > "$bad_rag_eval" <<'YAML' +version: "0.1" +project: + name: Demo + generated_at: "2026-05-24" + source_summary: + mode: explicit-files + refs: + - path: docs/current/architecture.md + note: current facts +cases: + - id: rag-20260524-001 + eval_type: citation_support + difficulty: medium + question: What supports this claim? + expected_answer_points: + - The answer should cite source evidence. + forbidden_claims: [] + tags: + - citation +YAML +if output=$(python3 "$SKILLS_DIR/knowject-rag-eval/scripts/validate-rag-eval-cases.py" "$bad_rag_eval" 2>&1); then + fail "uncited rag eval unexpectedly passed validation" +elif echo "$output" | grep -q "expected_source_refs"; then + pass "uncited rag eval reports validation error" +else + fail "uncited rag eval did not report expected source evidence error" +fi + +# 22. mcp-tool-designer schema example parses and covers read/write risks +note "mcp-tool-designer schema example parses" +mcp_schema_example="$SKILLS_DIR/knowject-mcp-tool-designer/references/examples/mcp-tools.schema.expected.json" +if [ -f "$mcp_schema_example" ]; then + if python3 -c " +import json +doc = json.load(open('$mcp_schema_example')) +tools = doc.get('tools') +assert isinstance(tools, list) and tools, 'tools must be non-empty list' +required = { + 'tool_name', + 'description', + 'source_endpoint', + 'input_schema', + 'output_shape_notes', + 'risk_level', + 'requires_confirmation', + 'auth_scope_notes', + 'audit_log_notes', + 'rollback_notes', +} +risk_levels = set() +for idx, tool in enumerate(tools): + missing = required - set(tool) + assert not missing, f'tools[{idx}] missing {sorted(missing)}' + src = tool['source_endpoint'] + assert isinstance(src, dict), f'tools[{idx}].source_endpoint must be object' + for key in ('method', 'path', 'source'): + assert isinstance(src.get(key), str) and src[key].strip(), f'tools[{idx}].source_endpoint.{key} required' + assert isinstance(tool['input_schema'], dict), f'tools[{idx}].input_schema must be object' + assert isinstance(tool['requires_confirmation'], bool), f'tools[{idx}].requires_confirmation must be boolean' + risk_levels.add(tool['risk_level']) +assert 'read_only' in risk_levels, 'example needs read_only tool' +assert risk_levels & {'low_write', 'destructive', 'external_side_effect', 'sensitive_data'}, 'example needs write/destructive/sensitive tool' +" 2>/dev/null; then + pass "mcp-tool-designer schema example validates" + else + fail "mcp-tool-designer schema example invalid" + fi +else + fail "mcp-tool-designer schema example missing" +fi + # Report echo "" echo "Checks: $checks | Failures: $failures"