Skip to content

Where do protocol test run records live? #44

Open

Description

@lwaldron

ADR 0016 (#43) separates a test run from a review. A review is a named person's judgement of a protocol; a run is a fact about one execution: agent, model, date, which tests passed. The checker writes a run record (run.yaml, format in PROTOCOL_STANDARD.md) and the person attaches it wherever it is useful, usually the PR or issue they are working on.

That is enough while there are few runs. It does not answer "which agents and models is this protocol known to work with", which is the question worth answering once runs accumulate. Options, roughly in order of weight:

  • A labelled issue per protocol in the content repository, run records posted as comments.
  • A testing-log repository in the federation, one file per run, indexed like PROTOCOLS.yaml.
  • A runs/ directory beside tests/ in the protocol directory. Rejected in the ADR: a commit and a PR for every run by every person, in a content repository, and runs conflated with the protocol's own history.

The benchmarking study in #5 is what will generate volume and should drive the choice. Nothing to decide until it does.

Drafted with Claude Code.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions