diff --git a/docs/26-skill-authoring-platform.md b/docs/26-skill-authoring-platform.md new file mode 100644 index 000000000..c77f4d288 --- /dev/null +++ b/docs/26-skill-authoring-platform.md @@ -0,0 +1,386 @@ +# Skill 创作与验证平台设计、部署与测试 + +## 定位 + +Skill 创作与验证平台让作者在 SkillHub 内完成 Skill 的**创建、编辑、受控验证和提交发布**, +而不是在仓库外部手工打包再上传。核心闭环是: + +```text +创建草稿 → 编辑 SKILL.md/资源文件 → 绑定 Agent/工具/MCP 运行时 + → 三层验证(结构/配置/行为,含日志与调用轨迹) + → 定位失败步骤 → 应用修复建议 → 复验 + → 达标草稿提交,复用既有扫描/审核/发布管线 +``` + +平台不引入第二套发布通道:提交动作直接复用 `SkillPublishService`,草稿只是把 +"写一个符合规范的 Skill" 变成了带验证门槛的结构化流程。 + +## 整体架构 + +```text +web/ React 工作台 + pages/authoring/drafts.tsx 草稿列表 + 创建 + pages/authoring/draft-detail.tsx 文件/运行时/运行记录三个标签页 + pages/authoring/run-detail.tsx 运行详情:事件控制台 + 发现列表 + features/authoring/… SSE hook、事件合并、diff 预览、发现卡片 + │ REST + SSE + ▼ +skillhub-app + controller/SkillAuthoringController 19 个 REST 端点 + controller/ValidationStreamController SSE 事件流(回放 + 直播) + service/authoring/ValidationRunOrchestrator 编排三层验证、事件、终态 + service/authoring/adapter/* LocalScript(inline/docker)/ OpenAiCompatible 运行时适配器 + service/authoring/mcp/* MCP 客户端(http/sse/stdio)与配置层连通探测 + service/authoring/ValidationEventBroadcaster SSE 广播 + ▼ +skillhub-domain + authoring/ SkillDraft、DraftFile、RuntimeBinding 实体与服务 + authoring/validation ValidationRun / ValidationEvent / ValidationFinding + authoring/spec validation.yaml 解析(防御式 SnakeYAML) + authoring/runtime 任务执行结果与断言求值器 + skill/validation/ SkillPackagePolicy(结构规则与发布侧共享,规则不漂移) + ▼ +skillhub-storage 草稿文件按内容寻址存入对象存储(与 Skill 包同机制) +``` + +## 领域模型与数据表 + +迁移 `V69__authoring_platform.sql` 新增六张表(初稿编号 V60,因上游 #874 统一身份 +平台占用了 V60–V65,rebase 后顺延为 V66;上游 promotion 与 system_auth_settings 又占用了 V66–V68,最终为 V69): + +| 表 | 职责 | +| --- | --- | +| `skill_draft` | 草稿元数据:归属人、名称、`revision`(每次内容变更 +1)、`contentDigest`、`validatedRevision`/`validatedRunId`、提交回填的 `submittedSkillId` | +| `draft_file` | 草稿内单个文件(路径、内容类型、对象存储 key、sha256) | +| `runtime_binding` | Agent 类型 + 适配器配置 + 工具白名单 + MCP 服务器声明(JSONB) | +| `validation_run` | 一次验证运行的快照:draft revision、状态、各层统计 | +| `validation_event` | 运行事件(日志、工具调用、阶段切换),`seq` 单调递增,供 SSE 回放 | +| `validation_finding` | 结构化发现:层级、规则码、严重度、`filePath`/位置、建议(JSONB) | + +关键不变量: + +- **乐观并发**:草稿保存文件带 `expectedRevision`,冲突返回 409,前端提示刷新后重试。 +- **验证即快照**:运行绑定启动时的 revision;`SUCCEEDED` 且 0 错误时回写 + `validatedRevision`。提交门槛是"当前 revision 必须恰好等于 `validatedRevision`", + 验证通过后再改动文件会自动失去提交资格,必须复验。 +- **运行状态机**:`QUEUED → PREPARING → RUNNING → SUCCEEDED / FAILED / CANCELLED / TIMED_OUT`; + 活跃运行支持协作式取消,超时由任务级 `timeoutMs` 与运行级 `run-timeout-ms` 双层控制, + 崩溃残留的活跃运行由维护任务按 `stale-run-minutes` 清扫为 `TIMED_OUT`。 + +## 三层验证流水线 + +`ValidationRunOrchestrator` 顺序执行三层,每层产出结构化发现而非异常字符串: + +1. **STRUCTURE(结构)**:`DraftStructureValidator` 检查包布局、路径合法性与去重、 + 扩展名白名单、内容与扩展名一致性、文件数与体积上限、SKILL.md frontmatter + 必填字段(name、description)。规则全部来自 `SkillPackagePolicy`,与发布侧 + 校验逐条对齐,草稿期能过的包到发布期不会再因结构被拒。 +2. **CONFIG(配置)**:`RuntimeBindingValidator` 校验 Agent 类型(`local-script` / + `openai-compatible`)、解释器白名单(`sh/bash/python3/node`)、工具白名单与 + MCP 声明(名称、transport、endpoint/command、工具过滤、env 引用),并扫描 + 绑定中任何凭据形态的字段(`CREDENTIAL_EMBEDDED`)——密钥只能存服务端配置。 + 同时对 validation.yaml 的全部断言做语法预检(类型合法、必填参数齐全、正则可编译), + 让错误在执行前暴露。声明的 MCP 服务器随后被**真实探测**:逐台连接、完成 + initialize 握手并 tools/list——连不上记 `MCP_CONNECT_FAILED` 错误(行为层 + 因此被阻断),`toolFilters` 引用服务器未提供的工具记 `MCP_TOOL_FILTER_UNKNOWN` + 告警,连接成功则在事件流留下工具清单日志。MCP 声明由此从"格式正确"变成 + "验证时确实可用"。 +3. **BEHAVIOR(行为)**:把草稿文件物化进一次性隔离工作区,按 validation.yaml + 逐任务执行(script 任务跑 `scripts/` 下的脚本,prompt 任务走 LLM 适配器), + 用 `AssertionEvaluator` 求值断言:`exit_code`、`stdout_contains`、`stdout_matches`、 + `stdout_json`(JSON Pointer 等值)、`artifact_exists`(可附 sha256)、 + `tool_call_count`,以及 prompt 侧的 `response_*` 系列。行为被真实执行验证, + 而不是靠元数据推断。 + +### 受控执行环境 + +`LocalScriptRuntimeAdapter` 是行为层默认运行时,有两种执行后端 +(`local-script.execution-mode`): + +- **inline**(默认,本地开发):脚本在服务器主机上以子进程执行。安全基线: + 每次运行独立临时工作区(`workspace-root` 下),脚本路径归一化后必须落在 + 工作区内,越界路径直接拒绝;进程环境被清空,只保留 `PATH`、`HOME` + (指向工作区)、`LANG`,外加绑定 `envAllowlist` 中显式点名的变量; + stdout/stderr 逐行流入事件流(作者实时可见),同时进入 256KB 截断缓冲用于 + 发现;解释器白名单外的命令不可执行;任务超时或运行取消时强制销毁进程。 +- **docker**(生产推荐):同一套脚本在锁定的容器内执行——`--network none` + (容器内无任何网络路由)、`--memory`/`--cpus`/`--pids-limit` 资源上限、 + 只读根文件系统(`--read-only`,工作区读写挂载到 `/workspace`、`/tmp` 为 + 限量 tmpfs)、`--cap-drop ALL` + `--security-opt no-new-privileges`、日志 + 驱动关闭。容器按运行标识命名,超时/取消时强制删除;Docker 不可用时该后端 + 报告不可用而不会静默回退。隔离效果由真实容器内的行为测试实证 + (`DockerScriptRuntimeAdapterTest`:路由表为空、根文件系统只读、工作区可写)。 + +`OpenAiCompatibleRuntimeAdapter` 处理 prompt 任务,是带工具执行的 agent 循环: +endpoint/model 来自绑定,API key 只从服务端 +`skillhub.authoring.openai-compatible.api-key` 解析,默认关闭。任务开始时连接 +绑定声明的每台 MCP 服务器并 tools/list,工具经服务器 `toolFilters` 与绑定全局 +`toolAllowlist`(空 = 不限制;条目匹配裸工具名或 `server.tool`)双重过滤后以 +function 形式暴露给模型;模型发起 tool_calls 时**真实调用**对应 MCP 服务器, +结果回填对话进入下一轮,轮数上限 `max-tool-rounds`(默认 4)。每轮对话、每次 +工具调用与结果都是事件流里的 `TOOL_CALL`/`TOOL_RESULT` 记录,`tool_call_count` +断言统计实际执行的工具调用数。 + +### 事件流(日志与调用轨迹) + +每个动作都是一条持久化事件:运行启停、阶段切换、任务工具调用(TOOL_CALL,含 +命令与参数)、任务结果、stdout/stderr 逐行日志、断言通过摘要。事件带单调 `seq`, +SSE 端点 `/api/web/authoring/runs/{runId}/events/stream` 支持从 `afterSeq` / +`Last-Event-ID` 回放再续播,终端运行的流会在回放后正常收尾,刷新页面不丢历史。 + +## 失败定位与修复建议闭环 + +发现(finding)携带层级、规则码、`filePath` + 位置(失败步骤定位)和可选的 +`FixSuggestion`:一个或多个 `FilePatch`(文件路径、旧内容 sha256、旧值、新值)。 +前端 `FindingCard` 渲染红绿 diff 预览,应用走两步确认(Apply → Confirm),后端 +`FindingFixService.apply` 校验文件未被并发修改后写入补丁并推进 `revision`; +`dismiss` 记录作者已知悉。修复后一键"Fix and re-validate"复验——示例 Skill 的 +实测:frontmatter 缺 `description` → `FRONTMATTER_FIELD_MISSING` 带定位与补丁 → +应用后 revision 2→3 → 复验 `SUCCEEDED`。 + +## 提交与发布管线衔接 + +`DraftSubmitService.submit` 的门槛:当前 revision 存在 0 错误的 `SUCCEEDED` 运行 +(告警已由验证过程向作者展示,视为已确认)。通过后把草稿文件打包交给 +`SkillPublishService`:重新校验包结构、触发安全扫描、进入既有审核流,草稿回填 +`submittedSkillId`/`submittedVersionId`。也就是说验证平台产出的是"已自证行为"的 +Skill 版本,扫描与审核沿用平台原有机制,没有旁路。 + +## API 一览 + +前缀 `/api/web/authoring`(同时挂 `/api/v1/authoring`): + +| 端点 | 说明 | +| --- | --- | +| `POST /drafts`、`GET /drafts`、`GET/DELETE /drafts/{id}` | 草稿 CRUD(创建时生成 SKILL.md 脚手架,validation.yaml 由作者按需在工作台新建) | +| `GET /drafts/{id}/files`、`PUT /drafts/{id}/files`、`GET /drafts/{id}/files/content`、`DELETE /drafts/{id}/files` | 文件列表 / 保存(带 expectedRevision)/ 读取 / 删除 | +| `GET/PUT /drafts/{id}/runtime` | 运行时绑定读写 | +| `POST /drafts/{id}/runs`、`GET /drafts/{id}/runs` | 启动验证 / 运行历史(版本与验证记录持久化) | +| `GET /runs/{id}`、`POST /runs/{id}/cancel` | 运行详情 / 取消活跃运行 | +| `GET /runs/{id}/events`、`GET /runs/{id}/events/stream` | 事件轮询 / SSE 流 | +| `GET /runs/{id}/findings`、`POST /runs/{id}/findings/{fid}/apply`、`POST /runs/{id}/findings/{fid}/dismiss` | 发现列表 / 应用修复 / 忽略 | +| `POST /drafts/{id}/submit` | 达标草稿提交发布 | + +## 前端工作台 + +路由(均需登录):`/dashboard/authoring`(列表)、`/dashboard/authoring/$draftId` +(文件 / 运行时 / 运行记录)、`/dashboard/authoring/$draftId/runs/$runId`(运行详情)。 + +- 文件编辑器支持新建/删除/保存,未保存标记 + 乐观并发冲突提示; +- 运行时表单按 Agent 类型切换字段(解释器 vs endpoint/model),工具白名单与 + MCP 服务器声明有即时校验; +- 运行详情页 = 状态卡 + 自动滚动事件控制台(SSE 直播,断线 2 秒轮询降级)+ + 发现卡片列表;活跃运行每 2 秒轮询状态直到终态; +- 文案覆盖 en/zh/ru 三语言。 + +## 部署 + +### 配置项(`skillhub.authoring.*`,均可经环境变量覆盖) + +| 配置 | 默认 | 说明 | +| --- | --- | --- | +| `workspace-root` | `${java.io.tmpdir}/skillhub-authoring` | 隔离工作区根目录(`SKILLHUB_AUTHORING_WORKSPACE_ROOT`) | +| `executor-threads` | 4 | 进程内并发验证线程数 | +| `run-timeout-ms` | 900000 | 单次运行硬上限(15 分钟) | +| `stale-run-minutes` | 30 | 崩溃恢复清扫阈值 | +| `local-script.enabled` | true | 本地脚本运行时开关 | +| `local-script.execution-mode` | **docker** | 脚本执行后端:`docker`(锁定容器,生产默认)/ `inline`(主机子进程,仅 local/dev/test profile;否则启动即失败,`SKILLHUB_AUTHORING_LOCAL_SCRIPT_MODE`) | +| `local-script.docker.image` / `memory` / `cpus` / `pids-limit` / `tmpfs-size` | alpine:3.20 / 256m / 1.0 / 128 / 64m | 容器隔离参数(`SKILLHUB_AUTHORING_DOCKER_*`);镜像需自带 Skill 用到的解释器 | +| `openai-compatible.enabled` | false | LLM 运行时开关(默认关闭) | +| `openai-compatible.api-key` | 空 | 服务端密钥,绝不写入草稿绑定 | +| `openai-compatible.default-endpoint` / `default-model` | 空 | prompt 任务的默认 LLM | +| `openai-compatible.max-tool-rounds` | 4 | 单个 prompt 任务的最大对话轮数(含 MCP 工具执行) | +| `openai-compatible.allow-private-endpoints` | false | LLM endpoint 的 SSRF 放行开关:私网 LLM(如内部 vLLM)才设 true(`SKILLHUB_AUTHORING_LLM_ALLOW_PRIVATE`) | +| `mcp.allow-private-endpoints` | false | http/sse MCP endpoint 的 SSRF 放行开关(`SKILLHUB_AUTHORING_MCP_ALLOW_PRIVATE`);链路本地/云元数据地址任何情况下都拒绝 | +| `mcp.stdio-enabled` | false | stdio MCP 传输总开关(`SKILLHUB_AUTHORING_MCP_STDIO_ENABLED`)——服务器侧进程执行,生产默认关闭 | +| `mcp.env-allowlist` | 空 | 允许 `envRefs` 引用的服务器环境变量名单(逗号分隔,`SKILLHUB_AUTHORING_MCP_ENV_ALLOWLIST`);默认空 = 全部拒绝 | +| `mcp.docker.image` / `memory` / `cpus` / `pids-limit` / `tmpfs-size` | alpine:3.20 / 256m / 1.0 / 128 / 64m | docker 模式下 stdio MCP 服务器的容器隔离参数(`SKILLHUB_AUTHORING_MCP_DOCKER_*`) | + +### 部署要点 + +- **数据库**:Flyway 迁移 V69 自动建表,无手工步骤;JSONB 列要求 PostgreSQL + (平台既有要求,无新增)。 +- **启用 prompt 任务**:配置 `SKILLHUB_AUTHORING_LLM_ENABLED=true` 并给出 + endpoint/model/api-key;不启用时含 prompt 任务的草稿会得到 `RUNTIME_DISABLED` + 配置层发现,script 流程不受影响。 +- **启用容器隔离**:`SKILLHUB_AUTHORING_LOCAL_SCRIPT_MODE=docker` 并确保运行节点 + 可访问 Docker daemon。镜像需包含 Skill 用到的解释器(alpine 只带 `sh`; + python3/node 的 Skill 应换基础镜像)。脚本执行不再依赖主机环境。 +- **MCP 服务器**:stdio 传输默认整体关闭(`mcp.stdio-enabled`),开启后命令在 + docker 执行模式下运行于与脚本相同的锁定容器(无网络、资源上限、只读根文件 + 系统,关闭时强制 `docker rm -f` 清理);inline 模式仅限 local/dev/test + profile。`envRefs` 只能引用 `mcp.env-allowlist` 点名的环境变量(默认空 = + 全部拒绝),这是凭据到达 MCP 服务器的唯一通道。http/sse 传输访问的 + endpoint 必须通过 SSRF 策略(见下)。验证时每台声明的服务器都会被真实探测。 +- **工作区**:验证运行会在 `workspace-root` 下创建一次性目录并在结束后清理, + 生产环境建议独立磁盘分区并纳入监控;多实例部署时运行在工作区所在节点本地 + 执行,无需共享存储。 +- **RISC-V64**:后端是架构无关的 Java 21 JAR,沿用 `skillhub-server` 既有 + `linux/riscv64` 构建路径(见 `docs/RISCV64.md`)。local-script 运行时只依赖 + 容器内 POSIX 工具(`sh`、`wc` 等),Temurin 运行时镜像自带;安全扫描器的 + RISC-V 限制与创作平台无关(扫描发生在提交后的发布管线)。 +- **安全基线**: + - **SSRF 防护**:MCP http/sse endpoint 与 OpenAI 兼容 endpoint 在保存时、 + 连接时双重校验——只允许 http(s),主机必须可解析,且解析到的地址不得为 + 链路本地(含云元数据 169.254.169.254)、组播或未指定地址(无条件拒绝), + 也不得为环回/RFC1918/ULA/CGNAT/RFC 2544 基准段 198.18.0.0/15(除非对应 + 表面的 `allow-private-endpoints` 放行;198.18.0.0/15 为保留段,也是 + fake-IP 代理/VPN 解析器的合成地址池)。HTTP 客户端从不跟随重定向(30x + 直接报错),单响应体上限 2MiB,stdio 单行上限 2MiB——阻断"公网 URL 302 + 跳元数据"与超大响应/超长行的内存耗尽攻击。已知边界:策略在检查时解析 + 域名,HttpClient 连接时再次解析,可缓解但无法完全消除 DNS rebinding + 竞态(绑定已校验地址的 socket 会破坏虚拟主机 endpoint,未实现)。 + - **生产强制容器**:`execution-mode` 默认 `docker`;若在非 local/dev/test + profile 下配置了 inline,应用启动即失败(`AuthoringStartupGuard`)。 + - **其余既有边界**:脚本在清空环境的最小工作区执行、解释器白名单、路径 + 不得越界、绑定禁止内嵌凭据、单文件 10MB / 包 100MB / 500 文件上限与 + 发布侧一致。 + +### RISC-V64 验证记录(2026-09-20) + +在 arm64 宿主机上用 Buildx + QEMU 对本功能构建 `linux/riscv64` 服务端镜像并完成 +活体验证(对 `docs/RISCV64.md` 组件镜像边界的实测补充): + +- **镜像构建**:`server/Dockerfile` 原生路径(JAR 在构建宿主侧编译,运行时基于 + `eclipse-temurin:21-jre-noble` 的 riscv64 官方变体),`docker image inspect` + 确认 `linux/riscv64`。 +- **启动**:QEMU 模拟下 Spring Boot 91.9 秒完成启动,`/actuator/health` 返回 + 200;Flyway 在全新 PostgreSQL 16 上完成全部迁移(含 V69 创作平台表)。 +- **创作平台全流程(REST API)**:创建草稿 → 读取脚手架 SKILL.md → 保存 + `scripts/check.sh` 与 `validation.yaml` → 保存 local-script 绑定 → 启动验证 → + 运行 `SUCCEEDED`(0 错误 0 警告),12 条事件完整落库且脚本 stdout 出现在事件 + 流中 —— 行为层的脚本子进程真实运行在 riscv64 用户态内。 +- **结论**:创作与验证平台不引入任何架构相关依赖(纯 JVM 字节码 + POSIX + 工具),随 `skillhub-server` 既有 `linux/riscv64` 发布路径交付即可; + `execution-mode: docker` 时基础镜像需有 riscv64 变体(默认 `alpine:3.20` + 官方提供)。 +- **边界**:本记录为 QEMU 模拟验证,与 CI 的 riscv64 镜像 guardrail 同级;原生 + RISC-V 硬件上的全栈冒烟(PostgreSQL/Redis/对象存储/scanner)仍按 + `docs/RISCV64.md` 的边界声明执行。 + +#### 复验脚本与原生硬件清单 + +上述流程已固化为 `scripts/riscv64-verify.sh`(自动识别宿主架构:riscv64 主机 +原生执行,其余走 QEMU;对独立数据库跑 创建→验证→SUCCEEDED→事件流检查)。 +拿到真机后,`scripts/riscv64-native-setup.sh` 可以一键完成环境准备:安装 +Docker/PostgreSQL/Redis(apt 与 dnf 系)、建空库、放通 docker 网桥、原生构建 +镜像并跑完验证,全过程落盘为 `riscv64-native-evidence-*.log`。注意原生构建 +需 `--build-arg BUILD_IMAGE=eclipse-temurin:21-jdk-noble`(Dockerfile 默认的 +Alpine JDK 构建镜像无 riscv64 变体,Noble 有)——setup 脚本已自动处理。 +在真实 RISC-V64 硬件或 CI runner 上完成原生验证时,除直接运行该脚本外, +按以下清单执行并留档: + +1. `uname -m` 输出 `riscv64`(非 QEMU:`/proc/cpuinfo` 无 "emodel"/QEMU 标记, + 且启动日志无 qemu 前缀); +2. PostgreSQL 16、Redis 7 以 riscv64 原生镜像运行(非 binfmt 转译); +3. `scripts/riscv64-verify.sh` 全部检查通过,记录 耗时 与 事件数 两个数字; +4. `execution-mode: docker` 下再跑一次(验证 alpine:3.20 riscv64 变体内的 + 脚本隔离执行); +5. 浏览器 E2E(`e2e/authoring-flow.spec.ts`)对准该实例跑通。 + +截至 2026-10-01,上述第 1–5 项中已有部分由**真机 CI**覆盖(见下节),第 4–5 +项仍待在独占真机环境补齐;PLCT / INCHI RVLab 的真机使用申请已提交 +(开源项目免费通道,开通周期约一周),获批后可补齐剩余项。 + +#### 真机 RISC-V64 CI 证据(2026-10-02,Cloud-V 板卡 runner) + +真机验证 workflow(`riscv64-native.yml`)运行在贡献者 fork +(`zjncs/skillhub`)的 [Actions](https://github.com/zjncs/skillhub/actions/workflows/riscv64-native.yml) +上——上游仓库没有板卡 runner,无法运行;复验用的 workflow 与脚本 +(`scripts/riscv64-native-setup.sh`、`scripts/riscv64-verify.sh`)随本 +文档交付,绿 run 日志即 RISC-V64 支持的证据。板卡由 Cloud-V / +10xEngineers 免费提供(物理 VisionFive 2,每次任务全新容器,无需账号 +或 token)。首次运行 +([run 37035583787](https://github.com/zjncs/skillhub/actions/runs/37035583787), +提交 `9f3451d2`)**全绿,全程约 14 分钟**: + +- **板卡原生性**:`Linux sf2-4 7.0.0-30-generic #30.1~24.04.1-Ubuntu … + riscv64 riscv64 riscv64 GNU/Linux`,`/proc/cpuinfo` 无 qemu 标记; +- **原生构建**:板卡上 apt 安装 openjdk-21 / postgresql / redis-server 后, + Maven 在 riscv64 上完成整个多模块构建; +- **验证流程**(`scripts/riscv64-verify.sh` jar 模式,7/7 通过):服务器在 + riscv64 用户态运行 → 健康检查通过 → 创建草稿 → 脚手架读取 → 绑定 → + 验证运行 `SUCCEEDED` → 脚本 stdout 出现在事件流(12 条事件)。 +- **全量后端测试在真机首跑(2026-10-08)**:1085 个测试中 1081 个通过, + 仅有的 4 个失败全部来自 `DockerScriptRuntimeAdapterTest`——即上文共享 + daemon 的 bind-mount 限制(环境问题,非架构问题)。该测试现已加入 + bind-mount 往返假设(与"无 Docker 时静默跳过"同一模式):在此类 + runner 上优雅跳过,在真实 Docker 环境照常执行(本地验证 4/4 通过)。 + 复跑确认:**`BUILD SUCCESS`,1081 个测试 0 失败 0 错误**(38 分 42 秒, + 4 个 docker 适配器测试按假设跳过),且整个 run 三 job 全绿 + ([run 37762771959](https://github.com/zjncs/skillhub/actions/runs/37762771959), + 含磁盘清理步骤以规避板卡收尾清理的磁盘压力问题)。——整个后端套件 + 在物理 RISC-V64 板卡上原生通过。 + +对照上面的原生清单:第 1、3 项已由此覆盖;第 2 项的平台服务在板卡上以 +riscv64 原生包运行(容器镜像的 riscv64 变体另行经 `docker manifest inspect` +验证:`postgres:16`/`alpine:3.20`/`eclipse-temurin:21-jre-noble` 均有, +`redis:7` 无);第 4 项(docker 执行模式重跑)在板卡 CI 上**不可验证**—— +Cloud-V 这类共享宿主 daemon 的 runner 会按宿主文件系统解析 bind-mount 源 +路径,job 容器内的工作区路径在宿主上不存在(预检往返测试留档: +`cat: can't open '/probe/probe.txt'`),docker 执行模式的验证以本地真实 +Docker 环境的证据为准(验证事件流记录 `backend: docker`、退出码 0,且 +`DockerScriptRuntimeAdapterTest` 在真实容器中实证隔离参数);第 5 项 +(浏览器 E2E)待独占真机环境补齐。该 workflow 带 `if: github.repository == 'zjncs/skillhub'` 守卫, +上游仓库无此 runner,不会被排队。 + +### 发布管线端到端证据(2026-09-21) + +`scripts/authoring-publish-e2e.sh` 在本地 dev 栈(PostgreSQL/Redis/MinIO + +skill-scanner :8000,`local` profile)上实跑全链路并留档,16/16 检查通过: + +- **创作与验证**(普通用户 `local-user`):草稿 94 → SKILL.md/`scripts/check.sh`/ + `validation.yaml` → local-script 绑定 → 验证运行 `SUCCEEDED`; +- **提交 PUBLIC**:submit 返回 skill 33 / version 33 / slug + `publish-evidence-1789957612` / 版本号 `20260921.102653`,版本进入 + `SCANNING`(非 SUPER_ADMIN 提交不会自动发布); +- **安全扫描**:异步完成后版本回到 `PENDING_REVIEW`,`security_audit` 落库 + `SKILL_SCANNER:SAFE:1`(1 条发现、判定安全); +- **人工审核**:平台管理员(`local-admin`,SUPER_ADMIN)通过 + `POST /api/web/reviews/1/approve` 批准,审核任务 `APPROVED`; +- **发布终态**:`skill_version.status = PUBLISHED`、`published_at` 已置位、 + `skill.visibility = PUBLIC`,另一普通用户(`local-plain-user`)可公开检索 + 该 Skill,且草稿回填 `submittedSkillId = 33`。 + +## 测试 + +| 层 | 位置 | 覆盖 | +| --- | --- | --- | +| 领域单测 | `skillhub-domain/…/authoring/` | `DraftStructureValidatorTest`(结构规则与修复建议)、`ValidationSpecParserTest`(防御式 YAML 解析、任务/断言语义)、`AssertionEvaluatorTest`(全部断言类型与失败信息)、`SkillScaffoldGeneratorTest` | +| 运行时单测 | `skillhub-app/…/authoring/adapter/` | `DockerScriptCommandBuilderTest`(隔离参数与挂载构造)、`DockerScriptRuntimeAdapterTest`(真实容器内实证:无网络路由、只读根文件系统、工作区可写、输出捕获;无 Docker 时静默跳过)、`OpenAiCompatibleRuntimeAdapterToolLoopTest`(agent 循环:MCP 工具发现→模型调用→真实执行→轨迹回填,toolFilters/toolAllowlist 过滤、轮数上限) | +| MCP 单测 | `skillhub-app/…/authoring/mcp/` | `HttpMcpClientTest`(initialize/tools 握手、会话头复用、SSE 帧解析、错误结果、不可达报错、**30x 重定向不跟随、2MiB 响应上限**,对真实本地 HTTP 服务器)、`StdioMcpClientTest`(stdio 传输对 python3 子进程;**2MiB 单行上限 + 进程销毁、关闭时清理命令执行**)、`McpProbeServiceTest`(连不上→`MCP_CONNECT_FAILED`、未知 toolFilter→告警、可跳过畸形声明)、`McpClientFactoryTest`(**连接时安全策略:被拒 endpoint/被禁 stdio 不建连不建进程、envRefs 白名单过滤、docker 包装参数**) | +| 安全策略单测 | `skillhub-app/…/config/` | `ConfiguredAuthoringSecurityPolicyTest`(SSRF 地址分类:云元数据/链路本地无条件拒绝、环回/RFC1918/ULA/CGNAT/198.18.0.0/15 按表面开关、公网放行、不可解析与 fake-IP 合成应答均拒绝——经注入 resolver,不依赖网络)、`AuthoringStartupGuardTest`(非 local/dev/test profile + inline → 启动失败) | +| 端到端集成 | `skillhub-app/…/authoring/AuthoringFlowIntegrationTest` | Testcontainers 真实 PostgreSQL 上跑通完整闭环:建草稿 → 改文件 → 绑定运行时 → 三层验证 → 修复发现 → 复验 → 提交(`ddl-auto=validate` 顺带校验 V69 与实体映射一致) | +| 前端单测 | `web/src/features/authoring/*.test.*` | 事件按 seq 合并去重、SSE 生命周期(回放合并、终态关闭、断线轮询降级)、修复 diff 预览、二进制文件 base64 处理 | +| 浏览器 E2E | `web/e2e/authoring-flow.spec.ts` | Playwright 真实 API 全 UI 闭环:创建草稿(命名空间/名称对话框)→ 文件编辑(SKILL.md/脚本/validation.yaml)→ 保存运行时绑定 → 启动验证至 Succeeded(事件控制台含脚本输出)→ 提交对话框过闸;坏 frontmatter → 失败发现 → diff 预览 → 两步确认应用修复 → 一键复验通过;二进制资源上传 → 只读面板(大小/类型/sha256)与字节级校验。断言落在持久 UI 状态(按钮态、徽章、响应体)而非易失 toast | +| 活体冒烟 | `scripts/authoring-smoke-test.sh` | 对运行中的服务器 35 项检查、四个场景:全层验证并提交;坏 frontmatter → 定位 → 应用修复 → 复验;守卫(未验证不可提交、跨用户不可读他人草稿);MCP 探测(死端点报 `MCP_CONNECT_FAILED`、本地假 MCP 服务器工具被发现并写入事件流、未知 toolFilter 仅告警),幂等可重复执行 | +| 示例 Skill | `examples/skill-authoring/` | 可直接装载的完整示例(含 4 个行为验证用例,覆盖全部 6 种 script 断言),README 给出 UI 与 API 两种装载方式 | + +常用命令: + +```bash +# 后端(需 Java 21;默认 JVM 24 会破坏 Mockito/ByteBuddy) +cd server && JAVA_HOME= ./mvnw -pl skillhub-app -am test + +# 前端 +cd web && pnpm lint && pnpm typecheck && pnpm test + +# 活体冒烟(先启动 dev profile 服务器) +./scripts/authoring-smoke-test.sh http://localhost:8082 + +# 浏览器 E2E(后端跑在 8082,vite 由 Playwright 拉起并把 /api 代理过去) +cd web && VITE_API_PROXY_TARGET=http://localhost:8082 pnpm test:e2e -- e2e/authoring-flow.spec.ts + +# 重新生成前端 API 类型 +cd web && pnpm exec openapi-typescript http://localhost:8082/v3/api-docs \ + -o src/api/generated/schema.d.ts +``` + +## 边界与非目标 + +- v1 的行为验证以确定性断言为主,语义质量评判不强行近似(归属模型辅助审核); +- 不支持草稿多人协作编辑(单一 owner + 乐观锁防误覆盖); +- 提交后草稿回填 `submittedSkillId`/`submittedVersionId` 作为回执,但并不锁定: + 继续编辑会推进 revision 并使已验证标记失效,再次提交前必须重新验证; +- MCP 工具调用发生在验证运行内(配置层探测 + prompt 任务执行),平台不充当 + 常驻的 MCP 网关;stdio MCP 命令与脚本任务共享同一台运行节点。 diff --git a/examples/skill-authoring/README.md b/examples/skill-authoring/README.md new file mode 100644 index 000000000..0da7af3a3 --- /dev/null +++ b/examples/skill-authoring/README.md @@ -0,0 +1,86 @@ +# Example skill: text-stats + +A complete, self-contained example for the Skill Authoring & Validation +platform. It demonstrates every artifact an author must produce and every +script assertion type the behavior layer can check. + +``` +examples/skill-authoring/ +├── SKILL.md # skill description: frontmatter (name, description) + usage docs +├── validation.yaml # behavior validation cases (4 tasks, 6 assertion types) +├── scripts/ +│ └── text-stats.sh # the executable helper: text file → JSON statistics +└── references/ + ├── metrics.md # background material the skill cites + └── sample.txt # fixture input with known counters (2 lines, 20 words) +``` + +The skill computes word/line/character statistics for a text file and emits a +single-line JSON object, which it also mirrors to `artifacts/report.json`: + +```json +{"input":"references/sample.txt","lines":2,"words":20,"characters":144} +``` + +## Validation cases + +| Task | What it proves | Assertion types used | +| -------------------- | ----------------------------------------------------- | -------------------- | +| `fixture-stats` | counters are correct for the bundled fixture | `exit_code`, `stdout_contains`, `stdout_json`, `tool_call_count` | +| `output-shape` | stdout is one JSON object with exactly the known keys | `exit_code`, `stdout_matches` | +| `report-artifact` | the run leaves a persistent report artifact | `exit_code`, `artifact_exists` | +| `rejects-missing-file` | bad usage fails loudly instead of printing garbage | `exit_code` | + +A valid run finishes `SUCCEEDED` with 4 tasks and 0 findings. To see the fix +loop in action, change the `/words` expectation to `21` and re-validate: the +run fails with an `ASSERTION_FAILED` finding pointing at the exact task. + +## Loading it into the platform + +### Option A: web workbench + +1. Sign in and open **Skill 创作** (Authoring) from the sidebar. +2. Create a draft named `text-stats` in any namespace you own. +3. In the **Files** tab, replace the scaffolded `SKILL.md`, then create + `validation.yaml`, `scripts/text-stats.sh`, and the two `references/` + files with the contents from this directory (paths matter, names are + up to you). +4. In the **Runtime** tab bind agent type **local-script** with interpreter + `sh`, no tool allowlist, no MCP servers. +5. Click **Validate** and watch the run stream in real time — structure, + config, then behavior events appear as they happen. +6. When the run succeeds, **Submit** pushes the draft into the existing + scan/review/publish pipeline. + +### Option B: REST API + +With the dev profile running (mock auth), the same flow is four calls — +see `scripts/authoring-smoke-test.sh` for a complete, runnable version +including CSRF bootstrapping: + +```sh +BASE=http://localhost:8082 +H='-H Content-Type:application/json -H X-Mock-User-Id:local-user' + +# 1. create the draft (id from the response) +curl -s $H -X POST $BASE/api/web/authoring/drafts \ + -d '{"namespaceSlug":"global","name":"text-stats","requirement":"text statistics as JSON"}' + +# 2. save each file (repeat for every path above) +curl -s $H -X PUT $BASE/api/web/authoring/drafts//files \ + -d '{"path":"SKILL.md","content":""}' + +# 3. bind the runtime +curl -s $H -X PUT $BASE/api/web/authoring/drafts//runtime \ + -d '{"agentType":"local-script","config":{"interpreter":"sh"},"toolAllowlist":[],"mcpServers":[]}' + +# 4. run validation, then poll the run / stream its events +curl -s $H -X POST $BASE/api/web/authoring/drafts//runs +curl -sN $BASE/api/web/authoring/runs//events/stream # live SSE +``` + +Scripts execute in an isolated workspace with a scrubbed environment (only +`PATH`, `HOME`, `LANG` plus variables named in the binding's `envAllowlist` +pass through), so the example deliberately sticks to `wc`/`tr`/`printf`, +which every Unix provides. Allowed interpreters are `sh`, `bash`, `python3`, +and `node`. diff --git a/examples/skill-authoring/SKILL.md b/examples/skill-authoring/SKILL.md new file mode 100644 index 000000000..e28583286 --- /dev/null +++ b/examples/skill-authoring/SKILL.md @@ -0,0 +1,33 @@ +--- +name: text-stats +description: computes word, line, and character statistics for a text file and reports them as JSON +--- + +# text-stats + +## Overview + +computes word, line, and character statistics for a text file and reports them as JSON. Agents use this skill whenever they need reproducible, machine-readable metrics about a piece of text — for example to compare a document before and after rewriting it. + +## Usage + +1. Locate the text file to analyze. Any plain-text file in the workspace works; `references/sample.txt` is a ready-made fixture. +2. Run the helper with the file path as the only argument: + + ```sh + sh scripts/text-stats.sh references/sample.txt + ``` + +3. Read the single-line JSON object from stdout: + + ```json + {"input":"references/sample.txt","lines":2,"words":20,"characters":144} + ``` + +4. The same object is mirrored to `artifacts/report.json` when a persistent copy is needed. + +## Resources + +- `references/metrics.md` — how each counter is defined +- `references/sample.txt` — sample input used by the validation cases +- `scripts/` — executable helpers invoked during task execution diff --git a/examples/skill-authoring/references/metrics.md b/examples/skill-authoring/references/metrics.md new file mode 100644 index 000000000..d5467bbb9 --- /dev/null +++ b/examples/skill-authoring/references/metrics.md @@ -0,0 +1,15 @@ +# Metric definitions + +All counters come from `wc(1)`, so results are reproducible across platforms +without extra dependencies. + +| Metric | Definition | +| ------------ | ---------------------------------------------------------------- | +| `lines` | Newline characters in the input (`wc -l`). A file whose last line has no trailing newline counts one line less. | +| `words` | Whitespace-separated tokens (`wc -w`). | +| `characters` | Bytes in the file (`wc -c`), so a multi-byte UTF-8 character counts once per byte. | + +The JSON object is emitted as a single line on stdout and mirrored to +`artifacts/report.json`. Invalid usage (wrong argument count or a missing +file) exits with code 2 and writes nothing, so a caller can distinguish +"computed" from "refused" without parsing error text. diff --git a/examples/skill-authoring/references/sample.txt b/examples/skill-authoring/references/sample.txt new file mode 100644 index 000000000..6d6112803 --- /dev/null +++ b/examples/skill-authoring/references/sample.txt @@ -0,0 +1,2 @@ +SkillHub makes agent skills discoverable, versioned, and trustworthy. +Every skill ships a validation spec so behavior is verified, not assumed. diff --git a/examples/skill-authoring/scripts/text-stats.sh b/examples/skill-authoring/scripts/text-stats.sh new file mode 100755 index 000000000..ab7148328 --- /dev/null +++ b/examples/skill-authoring/scripts/text-stats.sh @@ -0,0 +1,25 @@ +#!/bin/sh +# text-stats.sh — print word/line/character statistics for a text file as JSON. +# +# Usage: sh scripts/text-stats.sh +# Emits a single-line JSON object on stdout and mirrors it to artifacts/report.json. +set -eu + +if [ "$#" -ne 1 ]; then + echo "usage: text-stats.sh " >&2 + exit 2 +fi + +INPUT="$1" +if [ ! -f "$INPUT" ]; then + echo "input file not found: $INPUT" >&2 + exit 2 +fi + +LINES=$(wc -l < "$INPUT" | tr -d ' ') +WORDS=$(wc -w < "$INPUT" | tr -d ' ') +CHARS=$(wc -c < "$INPUT" | tr -d ' ') + +mkdir -p artifacts +printf '{"input":"%s","lines":%s,"words":%s,"characters":%s}\n' \ + "$INPUT" "$LINES" "$WORDS" "$CHARS" | tee artifacts/report.json diff --git a/examples/skill-authoring/validation.yaml b/examples/skill-authoring/validation.yaml new file mode 100644 index 000000000..d19e2970a --- /dev/null +++ b/examples/skill-authoring/validation.yaml @@ -0,0 +1,66 @@ +# Behavior validation cases for the text-stats skill. +# +# Every task runs the skill's own script inside the isolated validation +# workspace. Together the tasks exercise all six script assertion types: +# exit_code, stdout_contains, stdout_json, stdout_matches, artifact_exists, +# and tool_call_count. +version: 1 +tasks: + # Happy path: the fixture produces the documented counters. + - name: fixture-stats + description: the helper reports correct counters for references/sample.txt + type: script + script: scripts/text-stats.sh + args: ["references/sample.txt"] + timeoutMs: 15000 + assertions: + - type: exit_code + equals: 0 + - type: stdout_contains + value: '"input":"references/sample.txt"' + - type: stdout_json + pointer: /words + equals: 20 + - type: stdout_json + pointer: /lines + equals: 2 + - type: tool_call_count + min: 1 + max: 1 + + # Output shape: one JSON line with exactly the documented keys. + - name: output-shape + description: stdout is a single JSON object with the documented keys + type: script + script: scripts/text-stats.sh + args: ["references/sample.txt"] + timeoutMs: 15000 + assertions: + - type: exit_code + equals: 0 + - type: stdout_matches + pattern: '^\{"input":"[^"]+","lines":\d+,"words":\d+,"characters":\d+\}$' + + # Side effect: the report artifact is written to the workspace. + - name: report-artifact + description: the run leaves a machine-readable report artifact + type: script + script: scripts/text-stats.sh + args: ["references/sample.txt"] + timeoutMs: 15000 + assertions: + - type: exit_code + equals: 0 + - type: artifact_exists + path: artifacts/report.json + + # Failure mode: bad usage exits non-zero instead of printing partial output. + - name: rejects-missing-file + description: the helper refuses missing input files with a non-zero exit code + type: script + script: scripts/text-stats.sh + args: ["references/does-not-exist.txt"] + timeoutMs: 15000 + assertions: + - type: exit_code + equals: 2 diff --git a/scripts/authoring-publish-e2e.sh b/scripts/authoring-publish-e2e.sh new file mode 100755 index 000000000..7fc287623 --- /dev/null +++ b/scripts/authoring-publish-e2e.sh @@ -0,0 +1,259 @@ +#!/usr/bin/env bash + +set -euo pipefail + +# Publish-pipeline end-to-end evidence run for the authoring platform. +# +# Extends scripts/authoring-smoke-test.sh past the submit step: a validated +# draft is submitted as PUBLIC by a regular user, the security scanner +# (skillhub-skill-scanner-1) scans it asynchronously, a platform admin +# approves the review task, and the version reaches the PUBLISHED terminal +# state with the skill publicly retrievable. +# +# Each check prints PASS/FAIL plus the observed identifiers so the output can +# be pasted directly as evidence. Requires the local dev profile (mock auth, +# Postgres/Redis/scanner/minio from the compose stack). Usage: +# scripts/authoring-publish-e2e.sh [base-url] + +BASE_URL="${1:-http://localhost:8080}" +PASS=0 +FAIL=0 +COOKIE="$(mktemp)" +ADMIN_COOKIE="$(mktemp)" +RUN_ID="$(date +%s)" +NAME="publish-evidence-$RUN_ID" +PSQL=(docker exec skillhub-postgres-1 psql -U skillhub -d skillhub -tAc) + +cleanup() { + rm -f "$COOKIE" "$ADMIN_COOKIE" +} +trap cleanup EXIT + +pass() { + echo "PASS: $1" + PASS=$((PASS + 1)) +} + +fail() { + echo "FAIL: $1" + FAIL=$((FAIL + 1)) +} + +json_field() { + local json="$1" + local expr="$2" + JSON_INPUT="$json" python3 - "$expr" <<'PY' +import json +import os +import sys + +expr = sys.argv[1] +data = json.loads(os.environ["JSON_INPUT"]) +value = data +for part in expr.split('.'): + if part.isdigit(): + value = value[int(part)] + else: + value = value[part] +if isinstance(value, (dict, list)): + print(json.dumps(value, ensure_ascii=False)) +else: + print(value) +PY +} + +assert_equals() { + local description="$1" + local expected="$2" + local actual="$3" + if [[ "$expected" == "$actual" ]]; then + pass "$description" + else + fail "$description (expected '$expected', got '$actual')" + fi +} + +api() { + local method="$1" + local path="$2" + local data="${3:-}" + if [[ -n "$data" ]]; then + curl -sS -H "X-Mock-User-Id: local-user" -b "$COOKIE" -c "$COOKIE" \ + -H "X-XSRF-TOKEN: $CSRF" -H "Content-Type: application/json" \ + -X "$method" "$BASE_URL$path" -d "$data" + else + curl -sS -H "X-Mock-User-Id: local-user" -b "$COOKIE" -c "$COOKIE" \ + -H "X-XSRF-TOKEN: $CSRF" -X "$method" "$BASE_URL$path" + fi +} + +admin_api() { + local method="$1" + local path="$2" + local data="${3:-}" + if [[ -n "$data" ]]; then + curl -sS -H "X-Mock-User-Id: local-admin" -b "$ADMIN_COOKIE" -c "$ADMIN_COOKIE" \ + -H "X-XSRF-TOKEN: $ADMIN_CSRF" -H "Content-Type: application/json" \ + -X "$method" "$BASE_URL$path" -d "$data" + else + curl -sS -H "X-Mock-User-Id: local-admin" -b "$ADMIN_COOKIE" -c "$ADMIN_COOKIE" \ + -H "X-XSRF-TOKEN: $ADMIN_CSRF" -X "$method" "$BASE_URL$path" + fi +} + +await_run_terminal() { + local run_id="$1" + local status="" + for _ in $(seq 1 60); do + status="$(json_field "$(api GET "/api/web/authoring/runs/$run_id")" "data.status")" + case "$status" in + SUCCEEDED|FAILED|CANCELLED|TIMED_OUT) break ;; + esac + sleep 1 + done + echo "$status" +} + +await_db_value() { + local query="$1" + local expected="$2" + local timeout="${3:-60}" + local value="" + for _ in $(seq 1 "$timeout"); do + value="$("${PSQL[@]}" "$query" 2>/dev/null || true)" + [[ "$value" == "$expected" ]] && break + sleep 1 + done + echo "$value" +} + +echo "=== Authoring → Publish Pipeline E2E Evidence ===" +echo "Target: $BASE_URL Run: $RUN_ID" +echo + +curl -s -c "$COOKIE" -H "X-Mock-User-Id: local-user" "$BASE_URL/api/v1/auth/providers" >/dev/null +CSRF="$(awk '$6 == "XSRF-TOKEN" { print $7 }' "$COOKIE" | tail -n 1)" +curl -s -c "$ADMIN_COOKIE" -H "X-Mock-User-Id: local-admin" "$BASE_URL/api/v1/auth/providers" >/dev/null +ADMIN_CSRF="$(awk '$6 == "XSRF-TOKEN" { print $7 }' "$ADMIN_COOKIE" | tail -n 1)" + +# ---------------------------------------------------------------- author → validate + +echo "--- Stage 1: author and validate (regular user local-user) ---" + +DRAFT_RESPONSE="$(api POST /api/web/authoring/drafts "{\"namespaceSlug\":\"global\",\"name\":\"$NAME\"}")" +assert_equals "Draft created" "0" "$(json_field "$DRAFT_RESPONSE" "code")" +DRAFT_ID="$(json_field "$DRAFT_RESPONSE" "data.id")" +echo " draft id: $DRAFT_ID" + +SKILL_MD="--- +name: $NAME +description: publish pipeline evidence skill for run $RUN_ID +--- + +# Publish pipeline evidence + +Runs a shell check that must succeed before submission. +" +api PUT "/api/web/authoring/drafts/$DRAFT_ID/files" \ + "{\"path\":\"SKILL.md\",\"content\":$(python3 -c 'import json,sys; print(json.dumps(sys.stdin.read()))' <<< "$SKILL_MD"),\"contentType\":\"text/markdown\"}" >/dev/null +api PUT "/api/web/authoring/drafts/$DRAFT_ID/files" \ + "{\"path\":\"scripts/check.sh\",\"content\":\"#!/bin/sh\necho publish-evidence-ok\n\",\"contentType\":\"text/x-shellscript\"}" >/dev/null +api PUT "/api/web/authoring/drafts/$DRAFT_ID/files" \ + "{\"path\":\"validation.yaml\",\"content\":\"version: 1\\ntasks:\\n - name: check\\n description: script prints the evidence marker\\n type: script\\n script: scripts/check.sh\\n args: []\\n timeoutMs: 10000\\n assertions:\\n - type: exit_code\\n equals: 0\\n - type: stdout_contains\\n value: publish-evidence-ok\\n\",\"contentType\":\"application/yaml\"}" >/dev/null + +BINDING_RESPONSE="$(api PUT "/api/web/authoring/drafts/$DRAFT_ID/runtime" \ + '{"agentType":"local-script","config":{"interpreter":"sh"},"toolAllowlist":[]}')" +assert_equals "local-script binding saved" "0" "$(json_field "$BINDING_RESPONSE" "code")" + +RUN_RESPONSE="$(api POST "/api/web/authoring/drafts/$DRAFT_ID/runs")" +RUN_ID_LOCAL="$(json_field "$RUN_RESPONSE" "data.id")" +assert_equals "Validation run SUCCEEDED" "SUCCEEDED" "$(await_run_terminal "$RUN_ID_LOCAL")" + +# ---------------------------------------------------------------- submit PUBLIC + +echo +echo "--- Stage 2: submit as PUBLIC (enters publish pipeline) ---" + +SUBMIT_RESPONSE="$(api POST "/api/web/authoring/drafts/$DRAFT_ID/submit" '{"visibility":"PUBLIC"}')" +assert_equals "Draft submitted as PUBLIC" "0" "$(json_field "$SUBMIT_RESPONSE" "code")" +SKILL_ID="$(json_field "$SUBMIT_RESPONSE" "data.skillId")" +VERSION_ID="$(json_field "$SUBMIT_RESPONSE" "data.versionId")" +SKILL_VERSION="$(json_field "$SUBMIT_RESPONSE" "data.version")" +SLUG="$(json_field "$SUBMIT_RESPONSE" "data.slug")" +echo " skill id: $SKILL_ID version id: $VERSION_ID slug: $SLUG version: $SKILL_VERSION" + +DB_STATUS="$("${PSQL[@]}" "SELECT status FROM skill_version WHERE id = $VERSION_ID")" +if [[ "$DB_STATUS" == "PENDING_REVIEW" || "$DB_STATUS" == "SCANNING" ]]; then + pass "Version entered pipeline in $DB_STATUS (not auto-published)" +else + fail "Version should be PENDING_REVIEW/SCANNING, got '$DB_STATUS'" +fi + +# ---------------------------------------------------------------- security scan + +echo +echo "--- Stage 3: asynchronous security scan (scanner on :8000) ---" + +SCAN_STATUS="$(await_db_value \ + "SELECT status FROM skill_version WHERE id = $VERSION_ID" "PENDING_REVIEW" 90)" +assert_equals "Scan completed, version back to PENDING_REVIEW" "PENDING_REVIEW" "$SCAN_STATUS" + +SCAN_ROW="$("${PSQL[@]}" "SELECT scanner_type || ':' || verdict || ':' || findings_count FROM security_audit WHERE skill_version_id = $VERSION_ID AND deleted_at IS NULL ORDER BY id DESC LIMIT 1")" +if [[ -n "$SCAN_ROW" && "$SCAN_ROW" != "" ]]; then + pass "security_audit row recorded: $SCAN_ROW" +else + fail "No security_audit row for version $VERSION_ID" +fi + +REVIEW_TASK_ID="$("${PSQL[@]}" "SELECT id FROM review_task WHERE skill_version_id = $VERSION_ID ORDER BY id DESC LIMIT 1")" +if [[ -n "$REVIEW_TASK_ID" ]]; then + pass "Review task $REVIEW_TASK_ID pending approval" +else + fail "No review task created for version $VERSION_ID" +fi + +# ---------------------------------------------------------------- admin approve + +echo +echo "--- Stage 4: platform admin approves (local-admin) ---" + +APPROVE_RESPONSE="$(admin_api POST "/api/web/reviews/$REVIEW_TASK_ID/approve" \ + '{"comment":"publish-pipeline evidence run"}')" +assert_equals "Review approved" "0" "$(json_field "$APPROVE_RESPONSE" "code")" +REVIEW_STATUS="$(json_field "$APPROVE_RESPONSE" "data.status")" +assert_equals "Review task status APPROVED" "APPROVED" "$REVIEW_STATUS" + +# ---------------------------------------------------------------- terminal state + +echo +echo "--- Stage 5: PUBLISHED terminal state ---" + +DB_STATUS="$("${PSQL[@]}" "SELECT status FROM skill_version WHERE id = $VERSION_ID")" +assert_equals "skill_version.status = PUBLISHED (DB)" "PUBLISHED" "$DB_STATUS" + +PUBLISHED_AT="$("${PSQL[@]}" "SELECT published_at FROM skill_version WHERE id = $VERSION_ID")" +if [[ -n "$PUBLISHED_AT" && "$PUBLISHED_AT" != "" ]]; then + pass "published_at set: $PUBLISHED_AT" +else + fail "published_at not set" +fi + +VISIBILITY="$("${PSQL[@]}" "SELECT visibility FROM skill WHERE id = $SKILL_ID")" +assert_equals "skill.visibility = PUBLIC (DB)" "PUBLIC" "$VISIBILITY" + +DETAIL="$(curl -sS -H "X-Mock-User-Id: local-plain-user" "$BASE_URL/api/web/skills/global/$SLUG")" +assert_equals "Skill publicly retrievable by another user" "0" "$(json_field "$DETAIL" "code")" +assert_equals "Public detail shows visibility PUBLIC" "PUBLIC" "$(json_field "$DETAIL" "data.visibility")" + +SUBMITTED="$(api GET "/api/web/authoring/drafts/$DRAFT_ID")" +assert_equals "Draft shows submitted skill id $SKILL_ID" "$SKILL_ID" "$(json_field "$SUBMITTED" "data.submittedSkillId")" + +# ---------------------------------------------------------------- summary + +echo +if [[ "$FAIL" -eq 0 ]]; then + echo "=== All $PASS checks passed — full chain: create → validate → submit → scan → review → publish ===" + exit 0 +fi +echo "=== $FAIL check(s) failed, $PASS passed ===" +exit 1 diff --git a/scripts/authoring-smoke-test.sh b/scripts/authoring-smoke-test.sh new file mode 100755 index 000000000..c5e84f33a --- /dev/null +++ b/scripts/authoring-smoke-test.sh @@ -0,0 +1,511 @@ +#!/usr/bin/env bash + +set -euo pipefail + +# Authoring platform live smoke test. +# +# Drives the full draft lifecycle over HTTP against a running SkillHub server: +# case 1 — happy path: create draft → edit files → bind runtime → validate +# (structure + config + behavior) → SSE replay → submit +# case 2 — fix loop: broken SKILL.md → FAILED run → apply suggested fix → +# revision bump → re-validate → SUCCEEDED +# case 3 — guards: submit without validation and cross-user access are rejected +# case 4 — MCP probe: dead MCP endpoint → MCP_CONNECT_FAILED finding; a local +# fake MCP server → tools discovered in the event log; unknown +# toolFilters → warning without failing the run +# case 5 — security guards: cloud-metadata endpoints and envRefs outside the +# allowlist are rejected at binding save time +# +# Requires the local dev profile (mock auth enabled). Usage: +# scripts/authoring-smoke-test.sh [base-url] + +BASE_URL="${1:-http://localhost:8080}" +PASS=0 +FAIL=0 +COOKIE="$(mktemp)" +RUN_ID="$(date +%s)" +FAKE_MCP_PID="" +FAKE_MCP_PORT_FILE="$(mktemp)" + +cleanup() { + [[ -n "$FAKE_MCP_PID" ]] && kill "$FAKE_MCP_PID" 2>/dev/null || true + rm -f "$COOKIE" "$FAKE_MCP_PORT_FILE" +} +trap cleanup EXIT + +pass() { + echo "PASS: $1" + PASS=$((PASS + 1)) +} + +fail() { + echo "FAIL: $1" + FAIL=$((FAIL + 1)) +} + +csrf_token() { + awk '$6 == "XSRF-TOKEN" { print $7 }' "$COOKIE" | tail -n 1 +} + +bootstrap_csrf() { + curl -s -c "$COOKIE" -H "X-Mock-User-Id: local-user" "$BASE_URL/api/v1/auth/providers" >/dev/null +} + +json_field() { + local json="$1" + local expr="$2" + JSON_INPUT="$json" python3 - "$expr" <<'PY' +import json +import os +import sys + +expr = sys.argv[1] +data = json.loads(os.environ["JSON_INPUT"]) +value = data +for part in expr.split('.'): + if part.isdigit(): + value = value[int(part)] + else: + value = value[part] +if isinstance(value, (dict, list)): + print(json.dumps(value, ensure_ascii=False)) +else: + print(value) +PY +} + +assert_code() { + local description="$1" + local json="$2" + local expected="$3" + local actual + actual="$(json_field "$json" "code")" + if [[ "$actual" == "$expected" ]]; then + pass "$description" + else + fail "$description (expected code $expected, got $actual)" + fi +} + +assert_equals() { + local description="$1" + local expected="$2" + local actual="$3" + if [[ "$expected" == "$actual" ]]; then + pass "$description" + else + fail "$description (expected '$expected', got '$actual')" + fi +} + +api() { + local method="$1" + local path="$2" + local data="${3:-}" + if [[ -n "$data" ]]; then + curl -sS -H "X-Mock-User-Id: local-user" -b "$COOKIE" -c "$COOKIE" \ + -H "X-XSRF-TOKEN: $CSRF" -H "Content-Type: application/json" \ + -X "$method" "$BASE_URL$path" -d "$data" + else + curl -sS -H "X-Mock-User-Id: local-user" -b "$COOKIE" -c "$COOKIE" \ + -H "X-XSRF-TOKEN: $CSRF" -X "$method" "$BASE_URL$path" + fi +} + +# Polls a run until terminal; echoes the terminal status. +await_run_terminal() { + local run_id="$1" + local status="" + for _ in $(seq 1 60); do + status="$(json_field "$(api GET "/api/web/authoring/runs/$run_id")" "data.status")" + case "$status" in + SUCCEEDED|FAILED|CANCELLED|TIMED_OUT) break ;; + esac + sleep 1 + done + echo "$status" +} + +echo "=== Authoring Platform Smoke Test ===" +echo "Target: $BASE_URL" +echo + +bootstrap_csrf +CSRF="$(csrf_token "$COOKIE")" +if [[ -z "$CSRF" ]]; then + echo "Could not bootstrap CSRF token (is mock auth enabled?)" + exit 1 +fi + +# ---------------------------------------------------------------- fixture namespace + +# Discover an existing namespace the user can author in (namespace creation is +# admin-gated on the dev profile, so the smoke test reuses "global" or any +# namespace the mock user already owns). +NS_SLUG="$(api GET /api/web/me/namespaces | python3 -c 'import json,os,sys; data=json.loads(sys.stdin.read())["data"]; print(data[0]["slug"] if data else "")')" +if [[ -n "$NS_SLUG" ]]; then + pass "Authoring namespace available: $NS_SLUG" +else + fail "No accessible namespace for the mock user" + echo "=== $FAIL check(s) failed ===" + exit 1 +fi +SLUG="$NS_SLUG" + +# ---------------------------------------------------------------- case 1: happy path + +echo +echo "--- Case 1: draft validates across all layers and submits ---" + +# Draft names are unique per run so the smoke test is re-runnable against a +# shared dev database. +NAME="smoke-skill-$RUN_ID" +DRAFT_RESPONSE="$(api POST /api/web/authoring/drafts \ + "{\"namespaceSlug\":\"$SLUG\",\"name\":\"$NAME\",\"requirement\":\"smoke test skill\"}")" +assert_code "Draft created with scaffold" "$DRAFT_RESPONSE" "0" +DRAFT_ID="$(json_field "$DRAFT_RESPONSE" "data.id")" + +SKILL_MD="--- +name: $NAME +description: greets the caller with a fixed message +--- + +# Smoke skill + +## Overview + +greets the caller with a fixed message +" +FILES_RESPONSE="$(api PUT "/api/web/authoring/drafts/$DRAFT_ID/files" \ + "{\"path\":\"SKILL.md\",\"content\":$(python3 -c 'import json,sys; print(json.dumps(sys.stdin.read()))' <<< "$SKILL_MD"),\"contentType\":\"text/markdown\"}")" +assert_code "SKILL.md saved" "$FILES_RESPONSE" "0" + +SCRIPT_RESPONSE="$(api PUT "/api/web/authoring/drafts/$DRAFT_ID/files" \ + '{"path":"scripts/greet.sh","content":"echo hello from smoke\n"}')" +assert_code "Script file saved" "$SCRIPT_RESPONSE" "0" + +VALIDATION_YAML='version: 1 +tasks: + - name: greet + description: script prints the greeting + type: script + script: scripts/greet.sh + args: [] + timeoutMs: 10000 + assertions: + - type: exit_code + equals: 0 + - type: stdout_contains + value: hello from smoke +' +SPEC_RESPONSE="$(api PUT "/api/web/authoring/drafts/$DRAFT_ID/files" \ + "{\"path\":\"validation.yaml\",\"content\":$(python3 -c 'import json,sys; print(json.dumps(sys.stdin.read()))' <<< "$VALIDATION_YAML")}")" +assert_code "validation.yaml saved" "$SPEC_RESPONSE" "0" + +BINDING_RESPONSE="$(api PUT "/api/web/authoring/drafts/$DRAFT_ID/runtime" \ + '{"agentType":"local-script","config":{"interpreter":"sh"},"toolAllowlist":[],"mcpServers":[]}')" +assert_code "Runtime binding saved" "$BINDING_RESPONSE" "0" + +RUN_RESPONSE="$(api POST "/api/web/authoring/drafts/$DRAFT_ID/runs")" +assert_code "Validation run started" "$RUN_RESPONSE" "0" +RUN_ID="$(json_field "$RUN_RESPONSE" "data.id")" + +RUN_STATUS="$(await_run_terminal "$RUN_ID")" +assert_equals "Run reaches SUCCEEDED" "SUCCEEDED" "$RUN_STATUS" + +FINDINGS_RESPONSE="$(api GET "/api/web/authoring/runs/$RUN_ID/findings")" +assert_code "Findings readable" "$FINDINGS_RESPONSE" "0" +FINDING_COUNT="$(JSON_INPUT="$FINDINGS_RESPONSE" python3 -c 'import json,os; print(len(json.loads(os.environ["JSON_INPUT"])["data"]))')" +assert_equals "Zero findings on valid draft" "0" "$FINDING_COUNT" + +EVENTS_RESPONSE="$(api GET "/api/web/authoring/runs/$RUN_ID/events")" +assert_code "Events readable (polling API)" "$EVENTS_RESPONSE" "0" +EVENT_COUNT="$(JSON_INPUT="$EVENTS_RESPONSE" python3 -c 'import json,os; print(len(json.loads(os.environ["JSON_INPUT"])["data"]))')" +if [[ "$EVENT_COUNT" -gt 4 ]]; then + pass "Run produced $EVENT_COUNT events" +else + fail "Expected more than 4 events, got $EVENT_COUNT" +fi + +SSE_OUTPUT="$(curl -sN --max-time 10 -H "X-Mock-User-Id: local-user" -b "$COOKIE" \ + "$BASE_URL/api/web/authoring/runs/$RUN_ID/events/stream" || true)" +if grep -q "RUN_STARTED" <<< "$SSE_OUTPUT" && grep -q "RUN_FINISHED" <<< "$SSE_OUTPUT"; then + pass "SSE stream replays all events and terminates" +else + fail "SSE stream did not replay the full run" +fi + +DRAFT_AFTER="$(api GET "/api/web/authoring/drafts/$DRAFT_ID")" +assert_equals "Draft marked validated" "True" "$(json_field "$DRAFT_AFTER" "data.validated")" + +SUBMIT_RESPONSE="$(api POST "/api/web/authoring/drafts/$DRAFT_ID/submit" '{"visibility":"PRIVATE"}')" +assert_code "Validated draft submits to publish pipeline" "$SUBMIT_RESPONSE" "0" +SKILL_ID="$(json_field "$SUBMIT_RESPONSE" "data.skillId")" +if [[ "$SKILL_ID" != "None" && -n "$SKILL_ID" ]]; then + pass "Submit returned skill id $SKILL_ID" +else + fail "Submit did not return a skill id" +fi + +# ---------------------------------------------------------------- case 2: fix loop + +echo +echo "--- Case 2: broken draft gets a fixable finding and revalidates ---" + +BROKEN_NAME="broken-skill-$RUN_ID" +BROKEN_RESPONSE="$(api POST /api/web/authoring/drafts \ + "{\"namespaceSlug\":\"$SLUG\",\"name\":\"$BROKEN_NAME\"}")" +assert_code "Broken draft created" "$BROKEN_RESPONSE" "0" +BROKEN_ID="$(json_field "$BROKEN_RESPONSE" "data.id")" + +BROKEN_MD="--- +name: $BROKEN_NAME +--- + +# Broken +" +api PUT "/api/web/authoring/drafts/$BROKEN_ID/files" \ + "{\"path\":\"SKILL.md\",\"content\":$(python3 -c 'import json,sys; print(json.dumps(sys.stdin.read()))' <<< "$BROKEN_MD"),\"contentType\":\"text/markdown\"}" >/dev/null + +BROKEN_RUN="$(api POST "/api/web/authoring/drafts/$BROKEN_ID/runs")" +BROKEN_RUN_ID="$(json_field "$BROKEN_RUN" "data.id")" +assert_equals "Broken draft fails validation" "FAILED" "$(await_run_terminal "$BROKEN_RUN_ID")" + +BROKEN_FINDINGS="$(api GET "/api/web/authoring/runs/$BROKEN_RUN_ID/findings")" +FINDING_ID="$(JSON_INPUT="$BROKEN_FINDINGS" python3 -c ' +import json, os +findings = json.loads(os.environ["JSON_INPUT"])["data"] +match = next((f for f in findings if f["ruleCode"] == "FRONTMATTER_FIELD_MISSING"), None) +print(match["id"] if match else "none")')" +if [[ "$FINDING_ID" != "none" ]]; then + pass "FRONTMATTER_FIELD_MISSING finding reported with location" +else + fail "Expected FRONTMATTER_FIELD_MISSING finding" +fi + +REVISION_BEFORE="$(json_field "$(api GET "/api/web/authoring/drafts/$BROKEN_ID")" "data.revision")" +APPLY_RESPONSE="$(api POST "/api/web/authoring/runs/$BROKEN_RUN_ID/findings/$FINDING_ID/apply")" +assert_code "Suggested fix applied" "$APPLY_RESPONSE" "0" +REVISION_AFTER="$(json_field "$(api GET "/api/web/authoring/drafts/$BROKEN_ID")" "data.revision")" +if [[ "$REVISION_AFTER" -gt "$REVISION_BEFORE" ]]; then + pass "Applying fix advanced revision $REVISION_BEFORE → $REVISION_AFTER" +else + fail "Applying fix did not advance the revision" +fi + +FIXED_RUN="$(api POST "/api/web/authoring/drafts/$BROKEN_ID/runs")" +FIXED_RUN_ID="$(json_field "$FIXED_RUN" "data.id")" +assert_equals "Re-validation after fix succeeds" "SUCCEEDED" "$(await_run_terminal "$FIXED_RUN_ID")" + +# ---------------------------------------------------------------- case 3: guards + +echo +echo "--- Case 3: guards ---" + +UNVALIDATED_RESPONSE="$(api POST /api/web/authoring/drafts \ + "{\"namespaceSlug\":\"$SLUG\",\"name\":\"unvalidated-skill-$RUN_ID\"}")" +UNVALIDATED_ID="$(json_field "$UNVALIDATED_RESPONSE" "data.id")" +SUBMIT_UNVALIDATED="$(api POST "/api/web/authoring/drafts/$UNVALIDATED_ID/submit" '{"visibility":"PRIVATE"}')" +if [[ "$(json_field "$SUBMIT_UNVALIDATED" "code")" != "0" ]]; then + pass "Submit rejected for unvalidated draft" +else + fail "Submit should be rejected for an unvalidated draft" +fi + +OTHER_USER_COOKIE="$(mktemp)" +# A plain second user (no admin roles): local-admin holds SUPER_ADMIN in dev data +# and platform admins are allowed to read any draft by design. +curl -s -c "$OTHER_USER_COOKIE" -H "X-Mock-User-Id: local-plain-user" "$BASE_URL/api/v1/auth/providers" >/dev/null +OTHER_CSRF="$(awk '$6 == "XSRF-TOKEN" { print $7 }' "$OTHER_USER_COOKIE" | tail -n 1)" +CROSS_RESPONSE="$(curl -sS -H "X-Mock-User-Id: local-plain-user" -b "$OTHER_USER_COOKIE" \ + -H "X-XSRF-TOKEN: $OTHER_CSRF" "$BASE_URL/api/web/authoring/drafts/$DRAFT_ID")" +if [[ "$(json_field "$CROSS_RESPONSE" "code")" != "0" ]]; then + pass "Cross-user draft access rejected" +else + fail "Another user should not read someone else's draft" +fi +rm -f "$OTHER_USER_COOKIE" + +# ---------------------------------------------------------------- case 4: MCP probe + +echo +echo "--- Case 4: declared MCP servers are probed for real ---" + +MCP_NAME="mcp-skill-$RUN_ID" +MCP_RESPONSE="$(api POST /api/web/authoring/drafts \ + "{\"namespaceSlug\":\"$SLUG\",\"name\":\"$MCP_NAME\"}")" +assert_code "MCP probe draft created" "$MCP_RESPONSE" "0" +MCP_DRAFT_ID="$(json_field "$MCP_RESPONSE" "data.id")" + +MCP_MD="--- +name: $MCP_NAME +description: declares MCP servers on its runtime binding +--- + +# MCP probe skill +" +api PUT "/api/web/authoring/drafts/$MCP_DRAFT_ID/files" \ + "{\"path\":\"SKILL.md\",\"content\":$(python3 -c 'import json,sys; print(json.dumps(sys.stdin.read()))' <<< "$MCP_MD"),\"contentType\":\"text/markdown\"}" >/dev/null + +# Negative: an endpoint nothing listens on must surface as a config-layer error. +DEAD_BINDING="$(api PUT "/api/web/authoring/drafts/$MCP_DRAFT_ID/runtime" \ + '{"agentType":"local-script","config":{"interpreter":"sh"},"toolAllowlist":[],"mcpServers":[{"name":"dead-server","transport":"http","endpoint":"http://127.0.0.1:9/mcp"}]}')" +assert_code "Binding with dead MCP server accepted" "$DEAD_BINDING" "0" + +DEAD_RUN_ID="$(json_field "$(api POST "/api/web/authoring/drafts/$MCP_DRAFT_ID/runs")" "data.id")" +assert_equals "Dead MCP server fails the run" "FAILED" "$(await_run_terminal "$DEAD_RUN_ID")" + +DEAD_FINDINGS="$(api GET "/api/web/authoring/runs/$DEAD_RUN_ID/findings")" +DEAD_RULE="$(JSON_INPUT="$DEAD_FINDINGS" python3 -c ' +import json, os +findings = json.loads(os.environ["JSON_INPUT"])["data"] +match = next((f for f in findings if f["ruleCode"] == "MCP_CONNECT_FAILED"), None) +print(match["ruleCode"] if match else "none")')" +assert_equals "MCP_CONNECT_FAILED finding reported" "MCP_CONNECT_FAILED" "$DEAD_RULE" + +# Positive: a local fake MCP server; the probe must connect and list its tools. +FAKE_MCP_SCRIPT=' +import json +from http.server import BaseHTTPRequestHandler, HTTPServer + +class Handler(BaseHTTPRequestHandler): + def do_POST(self): + msg = json.loads(self.rfile.read(int(self.headers.get("Content-Length", 0)))) + method = msg.get("method", "") + if method == "initialize": + self.reply(msg, {"protocolVersion": "2024-11-05", "capabilities": {}, + "serverInfo": {"name": "smoke-mcp", "version": "1.0"}}, + session="smoke-session-1") + elif method == "tools/list": + self.reply(msg, {"tools": [{"name": "echo", "description": "echo text", + "inputSchema": {"type": "object"}}]}) + elif method == "tools/call": + self.reply(msg, {"content": [{"type": "text", "text": "ok"}]}) + else: + self.reply(msg, {}) + + def reply(self, msg, result, session=None): + body = json.dumps({"jsonrpc": "2.0", "id": msg.get("id"), "result": result}).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + if session: + self.send_header("Mcp-Session-Id", session) + self.end_headers() + self.wfile.write(body) + + def log_message(self, *args): + pass + +server = HTTPServer(("127.0.0.1", 0), Handler) +print(server.server_address[1], flush=True) +server.serve_forever() +' +python3 -c "$FAKE_MCP_SCRIPT" > "$FAKE_MCP_PORT_FILE" 2>/dev/null & +FAKE_MCP_PID=$! +MCP_PORT="" +for _ in $(seq 1 25); do + MCP_PORT="$(cat "$FAKE_MCP_PORT_FILE" 2>/dev/null || true)" + [[ -n "$MCP_PORT" ]] && break + sleep 0.2 +done +if [[ -n "$MCP_PORT" ]]; then + pass "Fake MCP server listening on port $MCP_PORT" +else + fail "Fake MCP server did not start" +fi + +LIVE_BINDING="$(api PUT "/api/web/authoring/drafts/$MCP_DRAFT_ID/runtime" \ + "{\"agentType\":\"local-script\",\"config\":{\"interpreter\":\"sh\"},\"toolAllowlist\":[],\"mcpServers\":[{\"name\":\"smoke-mcp\",\"transport\":\"http\",\"endpoint\":\"http://127.0.0.1:$MCP_PORT/mcp\",\"toolFilters\":[\"echo\"]}]}")" +assert_code "Binding with live MCP server accepted" "$LIVE_BINDING" "0" + +LIVE_RUN_ID="$(json_field "$(api POST "/api/web/authoring/drafts/$MCP_DRAFT_ID/runs")" "data.id")" +assert_equals "Live MCP server lets the run pass" "SUCCEEDED" "$(await_run_terminal "$LIVE_RUN_ID")" + +LIVE_EVENTS="$(api GET "/api/web/authoring/runs/$LIVE_RUN_ID/events")" +if grep -q "connected; tools: echo" <<< "$LIVE_EVENTS"; then + pass "Event log records discovered MCP tools" +else + fail "Event log lacks the MCP tools discovery line" +fi + +# toolFilters naming a tool the server does not expose is a warning, not an error. +FILTERED_BINDING="$(api PUT "/api/web/authoring/drafts/$MCP_DRAFT_ID/runtime" \ + "{\"agentType\":\"local-script\",\"config\":{\"interpreter\":\"sh\"},\"toolAllowlist\":[],\"mcpServers\":[{\"name\":\"smoke-mcp\",\"transport\":\"http\",\"endpoint\":\"http://127.0.0.1:$MCP_PORT/mcp\",\"toolFilters\":[\"echo\",\"does_not_exist\"]}]}")" +assert_code "Binding with unknown tool filter accepted" "$FILTERED_BINDING" "0" + +FILTERED_RUN_ID="$(json_field "$(api POST "/api/web/authoring/drafts/$MCP_DRAFT_ID/runs")" "data.id")" +assert_equals "Unknown tool filter still succeeds" "SUCCEEDED" "$(await_run_terminal "$FILTERED_RUN_ID")" + +FILTERED_FINDINGS="$(api GET "/api/web/authoring/runs/$FILTERED_RUN_ID/findings")" +FILTERED_RULE="$(JSON_INPUT="$FILTERED_FINDINGS" python3 -c ' +import json, os +findings = json.loads(os.environ["JSON_INPUT"])["data"] +match = next((f for f in findings if f["ruleCode"] == "MCP_TOOL_FILTER_UNKNOWN"), None) +print(match["ruleCode"] if match else "none")')" +assert_equals "MCP_TOOL_FILTER_UNKNOWN warning reported" "MCP_TOOL_FILTER_UNKNOWN" "$FILTERED_RULE" + +kill "$FAKE_MCP_PID" 2>/dev/null || true +wait "$FAKE_MCP_PID" 2>/dev/null || true +FAKE_MCP_PID="" + +# ---------------------------------------------------------------- case 5: security guards + +echo +echo "--- Case 5: binding security guards reject at save time ---" + +SEC_NAME="security-skill-$RUN_ID" +SEC_RESPONSE="$(api POST /api/web/authoring/drafts \ + "{\"namespaceSlug\":\"$SLUG\",\"name\":\"$SEC_NAME\"}")" +assert_code "Security guard draft created" "$SEC_RESPONSE" "0" +SEC_DRAFT_ID="$(json_field "$SEC_RESPONSE" "data.id")" + +SEC_MD="--- +name: $SEC_NAME +description: exercises the binding security policy +--- + +# Security guard skill +" +api PUT "/api/web/authoring/drafts/$SEC_DRAFT_ID/files" \ + "{\"path\":\"SKILL.md\",\"content\":$(python3 -c 'import json,sys; print(json.dumps(sys.stdin.read()))' <<< "$SEC_MD"),\"contentType\":\"text/markdown\"}" >/dev/null + +# Cloud metadata endpoints are always blocked — even on the local profile where +# loopback/RFC1918 endpoints are deliberately allowed. Reason text assertions +# live in AuthoringFlowIntegrationTest; here we pin the save-time rejection. +METADATA_BINDING="$(api PUT "/api/web/authoring/drafts/$SEC_DRAFT_ID/runtime" \ + '{"agentType":"local-script","config":{"interpreter":"sh"},"toolAllowlist":[],"mcpServers":[{"name":"metadata-thief","transport":"http","endpoint":"http://169.254.169.254/latest/meta-data/"}]}')" +if [[ "$(json_field "$METADATA_BINDING" "code")" != "0" ]]; then + pass "Cloud metadata MCP endpoint rejected at save" +else + fail "Cloud metadata MCP endpoint must be rejected at save" +fi + +# envRefs outside the (empty) allowlist are rejected: bindings must not read +# arbitrary server environment variables. +ENVREF_BINDING="$(api PUT "/api/web/authoring/drafts/$SEC_DRAFT_ID/runtime" \ + '{"agentType":"local-script","config":{"interpreter":"sh"},"toolAllowlist":[],"mcpServers":[{"name":"env-hog","transport":"http","endpoint":"http://127.0.0.1:9/mcp","envRefs":["HOME"]}]}')" +if [[ "$(json_field "$ENVREF_BINDING" "code")" != "0" ]]; then + pass "envRefs outside the allowlist rejected at save" +else + fail "envRefs outside the allowlist must be rejected at save" +fi + +# The LLM surface is guarded too — those requests would carry the server API key. +LLM_BINDING="$(api PUT "/api/web/authoring/drafts/$SEC_DRAFT_ID/runtime" \ + '{"agentType":"openai-compatible","config":{"endpoint":"http://169.254.169.254/v1","model":"test-model"},"toolAllowlist":[],"mcpServers":[]}')" +if [[ "$(json_field "$LLM_BINDING" "code")" != "0" ]]; then + pass "Cloud metadata LLM endpoint rejected at save" +else + fail "Cloud metadata LLM endpoint must be rejected at save" +fi + +# ---------------------------------------------------------------- summary + +echo +if [[ "$FAIL" -eq 0 ]]; then + echo "=== All $PASS checks passed ===" + exit 0 +fi +echo "=== $FAIL check(s) failed, $PASS passed ===" +exit 1 diff --git a/scripts/riscv64-native-setup.sh b/scripts/riscv64-native-setup.sh new file mode 100755 index 000000000..a6c76c5be --- /dev/null +++ b/scripts/riscv64-native-setup.sh @@ -0,0 +1,118 @@ +#!/usr/bin/env bash + +# One-shot native riscv64 verification setup for the SkillHub authoring +# platform. Runs ON the native riscv64 machine (e.g. a PLCT/INCHI RVLab +# board) after SSH access is granted: +# +# 1. installs Docker, PostgreSQL and Redis (Ubuntu/Debian via apt, +# openEuler via dnf); +# 2. creates the empty database the verification run migrates from zero +# (user skillhub / password skillhub_dev, matching the local profile); +# 3. opens PostgreSQL and Redis to the docker bridge so the server +# container can reach them (lab machine scope only); +# 4. builds the linux/riscv64 server image natively (no QEMU) — the +# Alpine JDK build stage has no riscv64 variant, so this passes the +# Noble one via --build-arg; +# 5. runs scripts/riscv64-verify.sh against the freshly built image and +# tees everything into an evidence log. +# +# Usage (from the skillhub checkout, as a user with sudo): +# sudo scripts/riscv64-native-setup.sh +# +# Evidence lands in riscv64-native-evidence-.log next to the script. +# Reviewers may additionally ask for: uname -a, /proc/cpuinfo head, and the +# absence of qemu strings in the boot log (native-proof items from +# docs/26-skill-authoring-platform.md). + +set -euo pipefail + +cd "$(dirname "$0")/.." +REPO_ROOT="$(pwd)" +DB_NAME="${SKILLHUB_RISCV_DB:-skillhub_riscv_check}" +BRIDGE_HOST="${SKILLHUB_RISCV_DB_HOST:-172.17.0.1}" +IMAGE="${SKILLHUB_RISCV_IMAGE:-skillhub-server:riscv64-native}" +EVIDENCE="riscv64-native-evidence-$(date +%Y%m%d-%H%M%S).log" + +log() { echo "$@" | tee -a "$EVIDENCE"; } + +[[ "$(uname -m)" == "riscv64" ]] || { echo "this script targets native riscv64 hosts; here uname -m is $(uname -m)" >&2; exit 1; } +[[ $EUID -eq 0 ]] || { echo "run with sudo (needs to install packages and edit service configs)" >&2; exit 1; } + +log "=== native riscv64 verification setup — $(date -u +%FT%TZ) ===" +log "host: $(uname -a)" +log "cpu: $(head -4 /proc/cpuinfo | tr '\n' ' ')" +if lsmod 2>/dev/null | grep -q qemu || grep -qi qemu /proc/cpuinfo; then + log "WARNING: qemu markers found — this may not be a native machine" +fi +log + +PKG="" +if command -v apt-get >/dev/null 2>&1; then + PKG=apt +elif command -v dnf >/dev/null 2>&1; then + PKG=dnf +else + echo "unsupported distro: need apt-get or dnf" >&2; exit 1 +fi +log "--- 1. install docker / postgresql / redis ($PKG) ---" +if [[ "$PKG" == "apt" ]]; then + export DEBIAN_FRONTEND=noninteractive + apt-get update -y + apt-get install -y docker.io postgresql redis-server +else + dnf install -y docker postgresql-server redis || dnf install -y docker podman postgresql-server redis +fi +systemctl enable --now docker 2>/dev/null || service docker start 2>/dev/null || true +systemctl enable --now redis 2>/dev/null || systemctl enable --now redis-server 2>/dev/null || service redis-server start 2>/dev/null || true +log "docker: $(docker --version 2>&1)" + +log "--- 2. prepare empty database $DB_NAME (Flyway migrates from zero) ---" +if [[ "$PKG" == "apt" ]]; then + PG_CONF=/etc/postgresql/*/main +else + PG_CONF=/var/lib/pgsql/data +fi +# listen on all interfaces and accept the docker bridge with password auth +if [[ "$PKG" == "apt" ]]; then + sed -i "s/^#\?listen_addresses.*/listen_addresses = '*'/" $PG_CONF/postgresql.conf + grep -q "172.17.0.0/16" $PG_CONF/pg_hba.conf || echo "host all all 172.17.0.0/16 scram-sha-256" >> $PG_CONF/pg_hba.conf + systemctl reload postgresql 2>/dev/null || systemctl restart postgresql +else + postgresql-setup --initdb 2>/dev/null || true + sed -i "s/^#\?listen_addresses.*/listen_addresses = '*'/" $PG_CONF/postgresql.conf + grep -q "172.17.0.0/16" $PG_CONF/pg_hba.conf || echo "host all all 172.17.0.0/16 scram-sha-256" >> $PG_CONF/pg_hba.conf + systemctl enable --now postgresql +fi +sudo -u postgres psql -v ON_ERROR_STOP=1 </dev/null | head -1) +if [[ -n "$REDIS_CONF" ]]; then + sed -i 's/^#\?bind .*/bind 127.0.0.1 172.17.0.1/' "$REDIS_CONF" + sed -i 's/^#\?protected-mode .*/protected-mode no/' "$REDIS_CONF" + systemctl restart redis 2>/dev/null || systemctl restart redis-server 2>/dev/null || service redis-server restart 2>/dev/null || true + log "redis listening on 127.0.0.1 + $BRIDGE_HOST" +else + log "redis.conf not found automatically — ensure redis answers on $BRIDGE_HOST:6379" +fi + +log "--- 4. build the image natively (no QEMU) ---" +docker build --build-arg BUILD_IMAGE=eclipse-temurin:21-jdk-noble \ + -t "$IMAGE" server/ 2>&1 | tee -a "$EVIDENCE" | tail -5 +docker image inspect "$IMAGE" --format 'built: {{.Os}}/{{.Architecture}}' | tee -a "$EVIDENCE" + +log "--- 5. run the authoring verification ---" +SKILLHUB_RISCV_DB="$DB_NAME" SKILLHUB_RISCV_DB_HOST="$BRIDGE_HOST" \ + bash scripts/riscv64-verify.sh "$IMAGE" 2>&1 | tee -a "$EVIDENCE" + +log +log "=== done — evidence saved to $EVIDENCE ===" diff --git a/scripts/riscv64-verify.sh b/scripts/riscv64-verify.sh new file mode 100755 index 000000000..03ef15cd4 --- /dev/null +++ b/scripts/riscv64-verify.sh @@ -0,0 +1,247 @@ +#!/usr/bin/env bash + +# RISC-V64 live verification of the authoring platform. +# +# Boots the linux/riscv64 server (native on riscv64 hosts, QEMU-emulated +# elsewhere) against a dedicated PostgreSQL database and drives the full +# authoring flow over the REST API: +# create draft → read scaffold → save script + validation.yaml → bind runtime +# → validate → SUCCEEDED → script stdout visible in the event stream. +# +# Boot modes (SKILLHUB_RISCV_BOOT_MODE, default docker): +# docker — run the linux/riscv64 image (see "Building the image" below); +# jar — run an already-built jar directly with java (no Docker needed; +# used by the real-board CI workflow, where the build itself proves +# the toolchain and PG/Redis run as services on the board). +# +# The script assumes: +# - docker mode: the image exists; PostgreSQL and Redis are reachable from +# the container (defaults use host.docker.internal on ports 5432/6379); +# - jar mode: SKILLHUB_RISCV_JAR exists (default +# server/skillhub-app/target/skillhub-app-0.1.0.jar) and PG/Redis answer +# on localhost; +# - the DB given by SKILLHUB_RISCV_DB is EMPTY — Flyway migrates it from zero +# (the script never touches your main dev database). +# +# Usage: +# scripts/riscv64-verify.sh [image] # default skillhub-server:riscv64 +# SKILLHUB_RISCV_DB=mydb scripts/riscv64-verify.sh myimage:tag +# SKILLHUB_RISCV_BOOT_MODE=jar SKILLHUB_RISCV_DB=mydb scripts/riscv64-verify.sh +# +# Building the image (on any host; buildx + QEMU required on non-riscv64): +# cd server && docker buildx build --platform linux/riscv64 \ +# -t skillhub-server:riscv64 --load . +# On a native riscv64 host pass --build-arg BUILD_IMAGE=eclipse-temurin:21-jdk-noble +# (the default Alpine JDK build stage has no riscv64 variant). +# +# Native hardware note: on a real riscv64 host (uname -m == riscv64) the same +# script runs without emulation; the log line marks which mode you are in. +# QEMU mode proves architecture independence of the JVM workload, native mode +# additionally proves the platform services' riscv64 images. + +set -u + +IMAGE="${1:-skillhub-server:riscv64}" +DB_NAME="${SKILLHUB_RISCV_DB:-skillhub_riscv_check}" +MODE="${SKILLHUB_RISCV_BOOT_MODE:-docker}" +JAR="${SKILLHUB_RISCV_JAR:-server/skillhub-app/target/skillhub-app-0.1.0.jar}" +HOST="${SKILLHUB_RISCV_DB_HOST:-$([[ "$MODE" == "jar" ]] && echo localhost || echo host.docker.internal)}" +CONTAINER="skillhub-riscv-check" +SERVER_PID="" +SERVER_LOG="/tmp/skillhub-riscv-verify.log" +BASE_URL="http://localhost:18082" +COOKIE="$(mktemp)" +PASS=0 +FAIL=0 + +pass() { echo "PASS: $1"; PASS=$((PASS + 1)); } +fail() { echo "FAIL: $1"; FAIL=$((FAIL + 1)); } + +json_field() { + JSON_INPUT="$1" python3 - "$2" <<'PY' +import json, os, sys +expr = sys.argv[1] +value = json.loads(os.environ["JSON_INPUT"]) +for part in expr.split('.'): + value = value[int(part)] if part.isdigit() else value[part] +print(json.dumps(value, ensure_ascii=False) if isinstance(value, (dict, list)) else value) +PY +} + +csrf_token() { awk '$6 == "XSRF-TOKEN" { print $7 }' "$COOKIE" | tail -n 1; } +bootstrap_csrf() { + curl -s -c "$COOKIE" -H "X-Mock-User-Id: local-user" "$BASE_URL/api/v1/auth/providers" >/dev/null +} +api() { + local method="$1" + local path="$2" + local body="${3:-}" + if [[ -n "$body" ]]; then + curl -s -b "$COOKIE" -c "$COOKIE" -X "$method" -H "Content-Type: application/json" \ + -H "X-Mock-User-Id: local-user" -H "X-XSRF-TOKEN: $(csrf_token)" -d "$body" "$BASE_URL$path" + else + curl -s -b "$COOKIE" -c "$COOKIE" -X "$method" \ + -H "X-Mock-User-Id: local-user" -H "X-XSRF-TOKEN: $(csrf_token)" "$BASE_URL$path" + fi +} + +cleanup() { + if [[ -n "$SERVER_PID" ]]; then + kill "$SERVER_PID" >/dev/null 2>&1 || true + fi + if [[ "$MODE" == "docker" ]]; then + docker rm -f "$CONTAINER" >/dev/null 2>&1 || true + fi + rm -f "$COOKIE" +} +trap cleanup EXIT + +echo "=== RISC-V64 authoring platform verification ===" +if [[ "$MODE" == "docker" ]]; then + echo "image: $IMAGE" +else + echo "jar: $JAR" +fi +echo "database: $DB_NAME on $HOST" +ARCH="$(uname -m)" +if [[ "$ARCH" == "riscv64" ]]; then + echo "mode: NATIVE riscv64 host" +else + echo "mode: QEMU emulation (host arch $ARCH)" +fi +echo + +echo "--- 1. boot the server (boot mode: $MODE) ---" +if [[ "$MODE" == "docker" ]]; then + docker rm -f "$CONTAINER" >/dev/null 2>&1 || true + docker run -d --name "$CONTAINER" -p 18082:8080 "$IMAGE" \ + --spring.profiles.active=local \ + --spring.datasource.url="jdbc:postgresql://$HOST:5432/$DB_NAME" \ + --spring.data.redis.host="$HOST" >/dev/null || { echo "docker run failed"; exit 1; } + IMAGE_ARCH="$(docker inspect "$CONTAINER" --format '{{.Image}}' >/dev/null 2>&1 && docker exec "$CONTAINER" uname -m 2>/dev/null || echo unknown)" +else + [[ -f "$JAR" ]] || { echo "jar not found at $JAR — build it or set SKILLHUB_RISCV_JAR"; exit 1; } + java -jar "$JAR" --spring.profiles.active=local --server.port=18082 \ + --spring.datasource.url="jdbc:postgresql://$HOST:5432/$DB_NAME" \ + --spring.data.redis.host="$HOST" > "$SERVER_LOG" 2>&1 & + SERVER_PID=$! + echo "server pid $SERVER_PID (log: $SERVER_LOG)" + IMAGE_ARCH="$(uname -m)" +fi +echo "arch the server runs on: $IMAGE_ARCH" +if [[ "$IMAGE_ARCH" == "riscv64" ]]; then + pass "server userland is riscv64" +else + fail "server userland is '$IMAGE_ARCH', expected riscv64" +fi + +HEALTHY=0 +for i in $(seq 1 90); do + code=$(curl -s -o /dev/null -w '%{http_code}' "$BASE_URL/actuator/health" 2>/dev/null || true) + if [[ "$code" == "200" ]]; then + echo "health 200 after ~$((i * 10))s" + HEALTHY=1 + break + fi + if [[ "$MODE" == "docker" ]]; then + if ! docker ps --format '{{.Names}}' | grep -q "^$CONTAINER$"; then + echo "CONTAINER DIED — last logs:" + docker logs "$CONTAINER" 2>&1 | tail -30 + exit 1 + fi + else + if ! kill -0 "$SERVER_PID" 2>/dev/null; then + echo "SERVER DIED — last logs:" + tail -30 "$SERVER_LOG" + exit 1 + fi + fi + sleep 10 +done +if [[ "$HEALTHY" == "1" ]]; then + pass "server boots and becomes healthy" +else + fail "server did not become healthy in 900s" + if [[ "$MODE" == "docker" ]]; then + docker logs "$CONTAINER" 2>&1 | tail -30 + else + tail -30 "$SERVER_LOG" + fi + exit 1 +fi + +echo "--- 2. authoring flow over the API ---" +bootstrap_csrf +RUN_TAG="$(date +%s)" +DRAFT="$(api POST /api/web/authoring/drafts \ + "{\"namespaceSlug\":\"global\",\"name\":\"riscv-verify-$RUN_TAG\",\"description\":\"verifies the authoring platform on riscv64\"}")" +if [[ "$(json_field "$DRAFT" "code")" == "0" ]]; then + pass "draft created" +else + fail "draft create: $DRAFT" + exit 1 +fi +DRAFT_ID="$(json_field "$DRAFT" "data.id")" + +SCAFFOLD="$(api GET "/api/web/authoring/drafts/$DRAFT_ID/files/content?path=SKILL.md")" +if echo "$SCAFFOLD" | grep -q "name: riscv-verify-$RUN_TAG"; then + pass "scaffold SKILL.md persisted and readable" +else + fail "scaffold read: $SCAFFOLD" +fi + +api PUT "/api/web/authoring/drafts/$DRAFT_ID/files" \ + '{"path":"scripts/check.sh","content":"#!/bin/sh\necho riscv-script-OK\n","contentType":"text/x-shellscript"}' >/dev/null +# generous task timeout: first docker execution-mode runs may pull the +# alpine:3.20 riscv64 image before the script can start +api PUT "/api/web/authoring/drafts/$DRAFT_ID/files" \ + '{"path":"validation.yaml","content":"version: 1\ntasks:\n - name: smoke\n description: script runs on riscv64\n type: script\n script: scripts/check.sh\n args: []\n timeoutMs: 180000\n assertions:\n - type: exit_code\n equals: 0\n - type: stdout_contains\n value: riscv-script-OK\n"}' >/dev/null + +BINDING="$(api PUT "/api/web/authoring/drafts/$DRAFT_ID/runtime" \ + '{"agentType":"local-script","config":{"interpreter":"sh"},"toolAllowlist":[],"mcpServers":[]}')" +if [[ "$(json_field "$BINDING" "code")" == "0" ]]; then + pass "runtime binding saved" +else + fail "binding: $BINDING" +fi + +RUN="$(api POST "/api/web/authoring/drafts/$DRAFT_ID/runs" '{}')" +RUN_ID="$(json_field "$RUN" "data.id")" + +STATUS="" +for _ in $(seq 1 60); do + STATUS="$(json_field "$(api GET "/api/web/authoring/runs/$RUN_ID")" "data.status")" + case "$STATUS" in + SUCCEEDED|FAILED|CANCELLED|TIMED_OUT) break ;; + esac + sleep 5 +done +if [[ "$STATUS" == "SUCCEEDED" ]]; then + pass "validation run SUCCEEDED (script executed in the server's riscv64 userland)" +else + fail "validation run ended as $STATUS" + # surface the findings so failures are diagnosable from the job log alone + FINDINGS="$(api GET "/api/web/authoring/runs/$RUN_ID/findings")" + JSON_INPUT="$FINDINGS" python3 -c ' +import json, os +for f in json.loads(os.environ["JSON_INPUT"])["data"]: + print(" finding: [%s/%s] %s" % (f.get("layer"), f.get("ruleCode"), f.get("message")))' 2>/dev/null || true +fi + +EVENTS="$(api GET "/api/web/authoring/runs/$RUN_ID/events")" +EVENT_COUNT="$(JSON_INPUT="$EVENTS" python3 -c ' +import json, os +print(len(json.loads(os.environ["JSON_INPUT"])["data"]))' 2>/dev/null || echo 0)" +if echo "$EVENTS" | grep -q "riscv-script-OK"; then + pass "script stdout present in the event stream ($EVENT_COUNT events)" +else + fail "script stdout missing from the event stream" +fi + +echo +if [[ "$FAIL" -eq 0 ]]; then + echo "=== All $PASS checks passed ===" + exit 0 +fi +echo "=== $FAIL check(s) failed, $PASS passed ===" +exit 1