feat(meta): add adopter pilot-report template and validator

.gitignore

@@ -60,4 +60,6 @@ __pycache__/
 campaign/
 campaigns/
 pilot-*/
+# tools/pilot-report-validator is a committed framework tool, not an evidence package
+!/tools/pilot-report-validator/
 reassess-*/

docs/issue-management/README.md

@@ -114,6 +114,15 @@ adopter's `<project-config>/` directory:
 **Experimental.** No adopter pilot has run an evaluation against
 this family yet. Shape may change between framework versions.
 
+To provide pilot feedback, copy
+[`docs/pilot-report-template.md`](../pilot-report-template.md) into your
+project notes, fill in each section, and optionally validate the filled-in
+report with:
+
+```bash
+uv run --project tools/pilot-report-validator pilot-report-validate <your-report.md>
+```
+
 ## Cross-references
 
 - [Top-level README — Adopting the framework](../../README.md#adopting-the-framework) — 3-step bootstrap.

docs/labels-and-capabilities.md

@@ -258,6 +258,7 @@ it implements multiple contracts (e.g. `tools/gmail` provides both
 | [`tools/skill-and-tool-validator`](../tools/skill-and-tool-validator/) | `substrate:framework-dev` | Skill-frontmatter and convention validator |
 | [`tools/spec-status-index`](../tools/spec-status-index/) | `substrate:framework-dev` | Index of spec / RFC implementation status — substrate that also doubles as a governance/stats view |
 | [`tools/spec-validator`](../tools/spec-validator/) | `substrate:framework-dev` | Spec-frontmatter and body-section validator — counterpart to `skill-and-tool-validator` for `tools/spec-loop/specs/` |
+| [`tools/pilot-report-validator`](../tools/pilot-report-validator/) | `substrate:framework-dev` | Adopter pilot-report validator — required frontmatter keys, no unfilled placeholders, valid profile, and required body sections; counterpart to `spec-validator` for `docs/pilot-report-template.md` |
 | [`tools/vcs`](../tools/vcs/) | `contract:source-control` | Backend-dispatching implementation of the source-control (VCS) capability ([`tools/github/source-control.md`](../tools/github/source-control.md)); complete Git backend plus detected extension points for non-Git VCS bridges (#601 Hg, #602 SVN) |
 
 A tool's capability is the **interface it provides**, not which skills

docs/mentoring/README.md

@@ -91,6 +91,15 @@ required key documentation.
 contributor-to-committer interaction path under evaluation conditions yet;
 shape may change between framework versions.
 
+To provide pilot feedback, copy
+[`docs/pilot-report-template.md`](../pilot-report-template.md) into your
+project notes, fill in each section, and optionally validate the filled-in
+report with:
+
+```bash
+uv run --project tools/pilot-report-validator pilot-report-validate <your-report.md>
+```
+
 ## Cross-references
 
 - [`MISSION.md` § Agentic Mentoring](../../MISSION.md#technical-scope) —

docs/pairing/README.md

@@ -94,6 +94,15 @@ project's tracker, label set, or any shared infrastructure.
 `skill-and-tool-validate`. No adopter-pilot evaluation has run yet;
 shape may change between framework versions.
 
+To provide pilot feedback, copy
+[`docs/pilot-report-template.md`](../pilot-report-template.md) into your
+project notes, fill in each section, and optionally validate the filled-in
+report with:
+
+```bash
+uv run --project tools/pilot-report-validator pilot-report-validate <your-report.md>
+```
+
 ---
 
 ## Cross-references

docs/pilot-report-template.md

@@ -0,0 +1,112 @@
+---
+skill: <skill-name>
+date: YYYY-MM-DD
+target_repo: <owner>/<repo>
+profile: asf
+reporter: <github-handle>
+---
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents**  *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [Pilot report template](#pilot-report-template)
+  - [Instructions](#instructions)
+- [Pilot report: \<skill-name\> on \<owner/repo\>](#pilot-report-skill-name-on-ownerrepo)
+  - [Skill or family](#skill-or-family)
+  - [Target repo and profile](#target-repo-and-profile)
+  - [Blocked preflights](#blocked-preflights)
+  - [False positives](#false-positives)
+  - [Confirmation points](#confirmation-points)
+  - [Privacy and adapter notes](#privacy-and-adapter-notes)
+  - [Proposed spec changes](#proposed-spec-changes)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+<!-- SPDX-License-Identifier: Apache-2.0
+     https://www.apache.org/licenses/LICENSE-2.0 -->
+
+# Pilot report template
+
+Copy this file into your project notes, fill in each section, and share
+it back with the framework maintainers (or keep it local).  Pilot
+reports are the primary feedback channel for moving a skill family from
+`experimental` to `stable`.
+
+To validate a filled-in report file, run:
+
+```bash
+uv run --project tools/pilot-report-validator pilot-report-validate <your-report.md>
+```
+
+---
+
+## Instructions
+
+1. Copy this file to a convenient location (e.g.
+   `<project-notes>/pilot-report-<skill>-<date>.md`).
+2. Fill in the frontmatter block at the very top of the file (keep it
+   at the top, above the table of contents, so the validator detects it)
+   and each section below.
+3. Replace placeholder text in *italics* with your findings.
+4. For any section where nothing applies, write: **None observed.**
+5. For proposed spec changes, reference the spec file path and section
+   where possible.
+
+---
+
+# Pilot report: \<skill-name\> on \<owner/repo\>
+
+## Skill or family
+
+*Which skill or skill family was piloted — e.g. `pairing-self-review`,
+`mentoring`, `repo-health`. If you ran a full family sweep, list each
+skill you exercised.*
+
+## Target repo and profile
+
+*The repository tested against (`owner/repo`) and the project profile
+used (`asf`, `non-asf`, or `custom`). If your project-config directory
+is public, link to it here.*
+
+## Blocked preflights
+
+*List any preflights the skill ran that blocked, were skipped, or
+produced a confusing result. Include the preflight name or description
+and why it triggered.*
+
+*If none: **None observed.***
+
+## False positives
+
+*Findings the skill surfaced that were incorrect, irrelevant, or
+misleading. Include the finding summary and why it was a false positive.
+Distinguish clearly between "wrong finding" and "right finding but
+unhelpful wording".*
+
+*If none: **None observed.***
+
+## Confirmation points
+
+*Steps where the skill prompted for maintainer confirmation before
+proceeding. Note any that felt misplaced — too early, too late, or
+absent when one was expected.*
+
+*If no issues: **All confirmation points felt appropriate.***
+
+## Privacy and adapter notes
+
+*Any Privacy-LLM gate activations, adapter mismatches, credential
+preflight issues, or unexpected external-content handling. Note which
+adapter path was exercised (e.g. GitHub Issues, GitHub PR, Gmail,
+PonyMail) and whether the data-not-instructions boundary held.*
+
+*If none: **None observed.***
+
+## Proposed spec changes
+
+*Specific changes to propose to the skill spec or adopter-contract docs.
+For each proposal, note the spec file path and section, the current
+wording, and the suggested change.*
+
+*If none: **No changes proposed at this time.***

docs/repo-health/README.md

@@ -9,6 +9,7 @@
     - [`dependency-audit` (experimental)](#dependency-audit-experimental)
     - [`license-compliance-audit` (experimental)](#license-compliance-audit-experimental)
     - [`flaky-test-triage` (experimental)](#flaky-test-triage-experimental)
+  - [Status](#status)
   - [Adopter contract](#adopter-contract)
   - [Cross-references](#cross-references)
 
@@ -132,6 +133,20 @@ to include or exclude.
 
 ---
 
+## Status
+
+**Experimental.** All five skills shipped. No adopter-pilot evaluation
+has run end-to-end yet; shape may change between framework versions.
+
+To provide pilot feedback, copy
+[`docs/pilot-report-template.md`](../pilot-report-template.md) into your
+project notes, fill in each section, and optionally validate the filled-in
+report with:
+
+```bash
+uv run --project tools/pilot-report-validator pilot-report-validate <your-report.md>
+```
+
 ## Adopter contract
 
 `projects/_template/repo-health-config.md` provides the per-project

pyproject.toml

@@ -109,6 +109,7 @@ members = [
   "tools/security-tracker-stats-dashboard",
   "tools/skill-and-tool-validator",
   "tools/skill-evals",
+  "tools/pilot-report-validator",
   "tools/spec-status-index",
   "tools/spec-validator",
   "tools/vcs",

tools/pilot-report-validator/README.md

@@ -0,0 +1,76 @@
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents**  *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [pilot-report-validator](#pilot-report-validator)
+  - [Prerequisites](#prerequisites)
+  - [What it checks](#what-it-checks)
+  - [Usage](#usage)
+  - [Writing a pilot report](#writing-a-pilot-report)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+<!-- SPDX-License-Identifier: Apache-2.0
+     https://www.apache.org/licenses/LICENSE-2.0 -->
+
+# pilot-report-validator
+
+**Capability:** substrate:framework-dev
+
+Validates adopter pilot-report files against the required schema defined
+in [`docs/pilot-report-template.md`](../../docs/pilot-report-template.md).
+Pilot reports are the feedback artefact that documents an adopter's
+end-to-end run of an experimental skill family, and are the primary
+evidence source for advancing a skill from `experimental` to `stable`.
+
+## Prerequisites
+
+- **Runtime:** Python 3.11+ run via `uv`; stdlib-only (no runtime
+  dependencies). The `dev` group pulls `pytest`, `ruff`.
+- **CLIs:** None beyond the runtime.
+- **Credentials / auth:** None.
+- **Network:** Runs fully offline against local report files.
+
+## What it checks
+
+For every `.md` file that carries a YAML frontmatter block:
+
+1. **Required frontmatter keys** — `skill`, `date`, `target_repo`,
+   `profile`.
+2. **Valid `profile` value** — `asf` | `non-asf` | `custom`.
+3. **No unfilled placeholders** — frontmatter values must not contain
+   un-substituted `<...>` tokens, and `date` must be a real ISO 8601
+   date (`YYYY-MM-DD`).
+4. **Required body sections** — `## Skill or family`,
+   `## Target repo and profile`, `## Blocked preflights`,
+   `## False positives`, `## Confirmation points`,
+   `## Privacy and adapter notes`, `## Proposed spec changes`.
+
+The frontmatter block must be at the very top of the file — YAML
+frontmatter placed lower in the document is not detected. Files without
+a top-of-file frontmatter block (e.g. README files) are silently
+skipped. `docs/pilot-report-template.md` ships with placeholder
+frontmatter values, so running the validator on the unedited template
+reports those placeholders until you fill them in.
+
+## Usage
+
+```bash
+# Validate a single filled-in report
+uv run --project tools/pilot-report-validator \
+    pilot-report-validate path/to/my-pilot-report.md
+
+# Validate every report in a directory
+uv run --project tools/pilot-report-validator \
+    pilot-report-validate path/to/reports/
+
+# Run the test suite (use --directory, not --project, to avoid the
+# duplicate-`tests`-package collision when run from the repo root)
+uv run --directory tools/pilot-report-validator --group dev pytest
+```
+
+## Writing a pilot report
+
+Copy `docs/pilot-report-template.md`, fill in the frontmatter and each
+section, then validate with this tool before sharing.  See the template
+for detailed per-section instructions.

tools/pilot-report-validator/pyproject.toml

@@ -0,0 +1,58 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "pilot-report-validator"
+version = "0.1.0"
+description = "Validate adopter pilot-report files — required frontmatter keys and body sections."
+readme = "README.md"
+requires-python = ">=3.11"
+license = { text = "Apache-2.0" }
+dependencies = []
+
+[project.scripts]
+pilot-report-validate = "pilot_report_validator:main"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/pilot_report_validator"]
+
+[tool.ruff]
+line-length = 110
+target-version = "py311"
+src = ["src", "tests"]
+
+[tool.ruff.lint]
+select = ["E", "W", "F", "I", "B", "UP", "SIM", "C4", "RUF"]
+ignore = ["E501"]
+
+[tool.ruff.lint.per-file-ignores]
+"tests/**" = ["B", "SIM"]
+
+[dependency-groups]
+dev = [
+  "pytest>=8.0",
+  "ruff>=0.4",
+]
+
+[tool.pytest.ini_options]
+minversion = "8.0"
+addopts = "-ra -q"
+testpaths = ["tests"]