Skip to content

test: publish a privacy-preserving independent benchmark reproduction kit #58

Description

@AjnasNB

User problem

Qarinah publishes reproducible local benchmark fixtures, but external developers need a standard way to test their own public or synthetic repositories without contributing private project history or accepting a marketing-only aggregate.

Proposed contract

Create a privacy-preserving reproduction kit that:

  • imports only user-selected public or synthetic evidence;
  • records fixture identity, command, package version, Node version, platform, estimator, and exact result hashes;
  • compares equivalent required current sources on both sides;
  • reports retrieval coverage, rank, conflicts, supersession, pack size, and estimated context reduction separately;
  • emits a signed or digest-addressed result bundle suitable for an issue attachment or independent archive;
  • refuses provider-cost, latency, and answer-quality conclusions unless those measurements are actually supplied.

Acceptance criteria

  • At least three redistributable software projects with distinct task shapes.
  • One command to reproduce each result from a clean install.
  • Machine-readable schema and human-readable report.
  • Negative controls for missing evidence, misleading similarity, private paths, and changed fixtures.
  • CI verifies arithmetic, source hashes, and result-to-version identity.
  • Contribution guide for independent results and responsible claim wording.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationenhancementNew feature or requesthelp wantedExtra attention is neededreleaseRelease, benchmark, or launch workroadmapPlanned beyond the initial foundation

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions