Skip to content

spec amend: generalize the unit-under-evaluation so agentic artifacts get a stable fingerprint (Principle VIII + FR-090/043/037) #44

Description

@eeee2345

Scenario the current spec produces a worse outcome for

When Foundry evaluates an agentic-system artifact — an MCP server manifest, a skill file, a tool manifest, an IO trace — there is no source function to name. FR-090's fingerprint tuple (normalized path, function/symbol, vulnerability class) cannot be formed, because the "function/symbol" coordinate has no referent for these targets.

The consequence isn't just that the target is out of scope; it's that findings on it become un-deduplicable. Two runs of the same finding on the same unchanged artifact produce no stable identity, so cross-run inheritance and issue-update-not-recreate both fail and every re-run re-files everything as new. That is exactly the SC-005 failure mode Principle VIII ("Fingerprints Are Stable Under Structure") exists to prevent. The principle as written is stricter than its own rationale requires: its goal is structural identity that survives edits, but its wording pins that structure to one target family (source functions), which produces a strictly worse outcome (no stable identity at all) for a target family that is increasingly common.

Agentic artifacts are a real and growing evaluation surface, and I don't think the fix is to bolt them onto exploratory mode as second-class findings — that's where dedup/evidence break. The clean fix is to let the structure coordinate generalize while keeping everything Principle VIII actually cares about (structural, edit-stable, no line numbers / snippets / timestamps / instance ids).

Principles it interacts with

  • VIII (primary) — the fingerprint's structural-identity requirement.
  • I (evidence) — artifact-located findings must be referenceable by the Evidence gate; flagged below as out of scope for a first change and left as a clarification.

Level

FR-level additions + one Principle scope-widen. No new core roles. Intended as MINOR on both constitution.md and spec.md. Source-code targets are unchanged; artifact support is opt-in behind [NEEDS CLARIFICATION].

Proposed diffs (for discussion)

1) constitution.md — Principle VIII, widen "structure" (MINOR, 0.2.0 → 0.3.0):

### VIII. Fingerprints Are Stable Under Structure

A finding's identity is its location in the system's structure, not its position
in time or text.

- For source-code targets: structure = (file path, function/method name, vulnerability class).
- For agentic-artifact targets: structure = (artifact type, artifact identifier, vulnerability class).

[NEEDS CLARIFICATION: how is the artifact identifier assigned so it is stable across
re-runs of an unchanged system and unique within it?]

Deduplication, cross-run inheritance, and issue-update-not-recreate all key on this
fingerprint. Line numbers, snippets, timestamps, and run/instance ids remain excluded.

2) spec.md — sibling FRs (source-function FRs unchanged; artifacts are opt-in analogs):

- FR-090a: For agentic-artifact targets, the fingerprint MUST be a deterministic hash of
  (artifact type, artifact identifier, vulnerability class), excluding temporal/instance fields.
- FR-043a: For agentic-artifact targets, location MUST record (artifact type, artifact
  identifier, version/deployment context).
- FR-037a: The Detector MAY apply rule-based checks to non-source-code targets; when it does,
  the per-unit-with-context requirement of FR-037 is preserved with the artifact's declared
  structure as the unit and its declared inputs/effects as the context.

3) spec.md — new §11.13 "Target abstraction": a [NEEDS CLARIFICATION]-gated contract for what an evaluation target is (source, artifact, or both), how artifacts are enumerated, and what metadata they carry. Source-code stays the default; artifact support is an explicit adopter choice.

Deliberately out of scope for this change (to keep it MINOR)

  • Mixed-target scans (source + artifact in one run) — needs Indexer (§5.2) / Cartographer to enumerate artifacts; left as a follow-up [NEEDS CLARIFICATION].
  • Evidence gate for artifacts (FR-087 analog) — left as a follow-up clarification.

This first change only makes an agentic artifact a legitimate unit with a stable identity, fingerprint, and location. Everything downstream that assumes source functions keeps working unchanged.

A concrete proposal for the open artifact identifier question

To keep the [NEEDS CLARIFICATION] from lingering, one scheme that has held up in production agentic scanners:

  • artifact type ∈ { mcp-server, skill, tool-manifest, io-trace }
  • artifact identifier = the artifact's canonical declared name/path in its own manifest or registry — e.g. for an MCP tool, <server>/<tool-name>; for a skill, the declared skill id; for a tool manifest, the manifest path.

This is edit-stable (survives body edits, breaks only on rename/move — same property Principle VIII wants) and unique within a system, so it drops straight into the tuple.

Version bumps

constitution.md 0.2.0 → 0.3.0 (MINOR, VIII widened) · spec.md 0.1.0 → 0.2.0 (MINOR, new FRs + §11.13) · Sync Impact Report updated · downstream re-check flagged for §5.2 (Indexer), §5.4 (Detector), §7.5 (Fingerprint), and principle-fr-matrix.

COI disclosure

I maintain an open-source agentic detection ruleset (MIT) that already targets these artifact types in production, and ATR has been merged into Cisco AI Defense's skill-scanner (#79, #99). I'm raising this as a spec-level gap, not to route the spec toward any one ruleset — the change above is format- and ruleset-neutral. If useful, I'm happy to contribute a non-exclusive worked example once the unit model lands, alongside the existing CodeGuard one.

Context: this supersedes the docs-only approach in #41, which @santosomar (and Thomas, per the review thread) correctly flagged as changing the target unit without changing the spec. Sequencing it here first, as suggested.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions