Skip to content

feat(provenance): content-derived Agent fingerprint for scoreboard - #198

Draft
zhangnju wants to merge 1 commit into
Prism-Shadow:mainfrom
zhangnju:feat/agent-provenance
Draft

feat(provenance): content-derived Agent fingerprint for scoreboard#198
zhangnju wants to merge 1 commit into
Prism-Shadow:mainfrom
zhangnju:feat/agent-provenance

Conversation

@zhangnju

@zhangnju zhangnju commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Add a reproducibility fingerprint to the self-evolution loop. Each scoreboard evaluation records only version + provider + model_id, with no content fingerprint, so editing the system prompt / AGENTS.md / a skill without bumping version leaves two evaluations looking identical while testing different Agents.

  • core: buildAgentProvenance() hashes the raw config inputs (system prompt template, AGENTS.md, tool contract, each installed SKILL.md) into per-part sub-hashes plus a top-level agent_sha256; canonical-JSON hashing for stability.
  • server: BenchmarkEvaluation gains an optional provenance field, parsed fault-tolerantly (snake_case on disk -> camelCase DTO; a block missing agent_sha256 is dropped whole).
  • cli: penguin provenance prints the fingerprint deterministically (yaml/json) -- the capture path the skills call, since an LLM cannot compute sha256.
  • skills: agent-optimization captures the fingerprint via the CLI before appending an evaluation; agent-evaluation notes the version tie-in.
  • example benchmark carries illustrative provenance so the eval center shows it.

Add a reproducibility fingerprint to the self-evolution loop. Each scoreboard
evaluation records only version + provider + model_id, with no content
fingerprint, so editing the system prompt / AGENTS.md / a skill without bumping
version leaves two evaluations looking identical while testing different Agents.

- core: buildAgentProvenance() hashes the raw config inputs (system prompt
  template, AGENTS.md, tool contract, each installed SKILL.md) into per-part
  sub-hashes plus a top-level agent_sha256; canonical-JSON hashing for stability.
- server: BenchmarkEvaluation gains an optional provenance field, parsed
  fault-tolerantly (snake_case on disk -> camelCase DTO; a block missing
  agent_sha256 is dropped whole).
- cli: `penguin provenance` prints the fingerprint deterministically (yaml/json)
  -- the capture path the skills call, since an LLM cannot compute sha256.
- skills: agent-optimization captures the fingerprint via the CLI before
  appending an evaluation; agent-evaluation notes the version tie-in.
- example benchmark carries illustrative provenance so the eval center shows it.
@hiyouga
hiyouga marked this pull request as draft August 7, 2026 08:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant