Skip to content

State where requirements on input materials live, and whether a violation is an error or a warning #46

Closed

Description

@lwaldron

Reviewing waldronlab/agent-protocols#53 (Shannon diversity), the rule about rarefaction ended up in three places: a Materials bullet, a sentence in Step 2, and a Notes paragraph. None of them says what an agent should do when it gets a non-rarefied count table. That's not the author's fault; the standard doesn't say where input requirements belong or what happens when one fails.

What the standard currently says about Materials is that the heading must exist, plus a recommended skeleton with Software & Repositories and Databases & Reference Data. Steps may contain anything. The only error/warning vocabulary is in the protocol-runner skill and is about protocol status, not inputs. Diagnostic tests refer to "a requirement the protocol states" without saying where it's stated, and can only check refusal (expected_absent), so there's no way to express "proceed, but warn". Existing protocols have improvised: constraints in Materials prose, then a "Step 1: Validate" that restates them.

Proposal, three short additions:

  1. Placement rule, under Materials in PROTOCOL_STANDARD.md:

    Materials lists every input the protocol consumes (data, parameters, software, reference data) and every condition an input must or should satisfy. Steps consume materials; they don't introduce new conditions on inputs. Statements about what the protocol does or doesn't do belong in the overview or Notes, not in Materials.

  2. Two severity tiers, using RFC 2119 words:

    A condition written with must is a requirement: an executor that finds it violated stops before Step 1, reports which condition failed, and produces none of the protocol's outputs. A condition written with should is an advisory: the executor proceeds, warns the user naming the condition and its consequence, and records the warning in the execution metadata.

    This applies to conditions written in a protocol's Materials section, not to the standard's own prose, which already uses these words in their ordinary sense. Because checking happens before Step 1, protocols stop needing a hand-written validation step for anything already in Materials. Checks that only arise during computation (a count sample whose total is zero) stay in the step where they arise.

  3. Matching hooks in the two places that execute or test protocols:

    • skills/protocol-runner/SKILL.md: a step before Step 1 that checks every Materials condition; must stops, should warns and continues.
    • test.yaml diagnostic mode: add expected_warning (a substring or pattern) alongside expected_absent, so a test can assert both the results and the warning.

Under this rule the Shannon example becomes one bullet:

- **Input table:** CSV, one row per feature, one column per sample. The first column
  must contain unique, non-missing feature identifiers. All other cells must be
  non-negative numbers with no missing values. When `Input scale` is `counts`, samples
  should be rarefied or otherwise normalized to a common sequencing depth; if they are
  not, warn that Shannon values aren't comparable across samples and advise rarefaction.

and the Step 2 sentence and Notes paragraph go away.

I'd leave the validator alone; it can't check prose semantics and this is enforceable in review. Related: #15, since the before-Step-1 check is part of the execution contract.

If this direction is right I'll write it up as an ADR.

Drafted with Claude Code; decisions and edits are mine.

Activity

  1. added a commit that references this issue on Oct 23, 2026
    6a04575
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions