Problem
SkillRoster's existing EvidenceQuality (observed, inferred, unknown) correctly describes the provenance and completeness of a reported fact. It does not describe whether a Skill's declared capability was checked, nor whether a particular check passed.
Codex exposes two materially different validation signals:
skill-creator's quick_validate.py checks SKILL.md structure only: frontmatter, name, description, and unfinished scaffold placeholders. It explicitly does not prove good decisions.- Its independent forward-testing guidance evaluates a Skill against a realistic request and observable invariants in an isolated workspace.
SkillRoster's frozen Codex routing experiments make the distinction even sharper: retrieval/load, task oracle, workspace safety, transcript integrity, and control validity can pass or fail independently. A single aggregate score would conceal those facts.
Decision Needed
Define a validation-evidence contract before adding runtime behavior. A record must keep these axes separate:
- source evidence quality: the existing
observed / inferred / unknown model;
- validation scope: for example
structure, declared_deterministic_contract, or an explicitly frozen isolated_behavior suite;
- execution state:
passed, failed, not_run, blocked, or unknown;
- reproducibility facts: subject content digest, checker/oracle identity and version, execution time, and scoped diagnostics or artifact reference.
No scalar health score, compliance percentage, or implied semantic quality follows from a passing check.
Proposed First Gate
This issue is design-only. It must decide:
- whether validation declarations require a SkillRoster-specific sidecar rather than unsupported
SKILL.md frontmatter;
- which declarations are safe to inspect read-only and which require an explicitly authorized isolated execution;
- how
not_run, missing declarations, stale digests, and inaccessible content remain distinct from failure;
- which existing Codex cold-routing harness receipt fields can be reused rather than creating a second evaluator.
A runtime implementation should begin only after a repeated, classified user need justifies one narrow validation family and its public JSON contract.
Non-goals
- Executing any Skill-provided script during Scan or Report.
- Treating structural validity as capability, safety, trust, or task-success proof.
- Adding an aggregate health score or compliance percentage.
- Changing Plan, Apply, Undo, routing rank, or Agent activation.
- Replacing existing frozen Codex or Pi harnesses.
Problem
SkillRoster's existing
EvidenceQuality(observed,inferred,unknown) correctly describes the provenance and completeness of a reported fact. It does not describe whether a Skill's declared capability was checked, nor whether a particular check passed.Codex exposes two materially different validation signals:
skill-creator'squick_validate.pychecksSKILL.mdstructure only: frontmatter, name, description, and unfinished scaffold placeholders. It explicitly does not prove good decisions.SkillRoster's frozen Codex routing experiments make the distinction even sharper: retrieval/load, task oracle, workspace safety, transcript integrity, and control validity can pass or fail independently. A single aggregate score would conceal those facts.
Decision Needed
Define a validation-evidence contract before adding runtime behavior. A record must keep these axes separate:
observed/inferred/unknownmodel;structure,declared_deterministic_contract, or an explicitly frozenisolated_behaviorsuite;passed,failed,not_run,blocked, orunknown;No scalar health score, compliance percentage, or implied semantic quality follows from a passing check.
Proposed First Gate
This issue is design-only. It must decide:
SKILL.mdfrontmatter;not_run, missing declarations, stale digests, and inaccessible content remain distinct from failure;A runtime implementation should begin only after a repeated, classified user need justifies one narrow validation family and its public JSON contract.
Non-goals