lwaldron · Oct 23, 2026 · Oct 23, 2026 · Oct 23, 2026 · Oct 23, 2026
@@ -158,6 +158,23 @@ The markdown body must contain a `## Materials` section and a `## Steps` section
 `### Step` heading, and `## History & Reviews` must be the last section. A document with metadata but no
 materials and no steps is not a protocol, however complete its frontmatter is, and CI rejects it.
 
+`## Materials` lists every input the protocol consumes (data, parameters, software, reference data)
+and every condition an input must or should satisfy. Steps consume materials; they do not introduce
+new conditions on inputs. Statements about what the protocol does or does not do belong in the
+overview or `## Notes`, not in Materials.
+
+A condition on a material is written with **must** or **should**, in the sense of RFC 2119, and an
+executor checks every condition before Step 1:
+
+*   A condition written with **must** is a requirement. An executor that finds it violated stops,
+    reports which condition failed, and produces none of the protocol's outputs.
+*   A condition written with **should** is an advisory. The executor proceeds, warns the user naming
+    the condition and its consequence, and records the warning with the execution.
+
+This applies to conditions written in a protocol's Materials section. The standard's own prose uses
+these words in their ordinary sense. A check that can only be made during computation, such as a
+sample whose total is zero, belongs in the step where it arises.
+
 The remaining structure is a recommendation rather than a requirement, and follows it for compatibility
 with future export tools such as protocols.io integration:
 
@@ -168,6 +185,11 @@ Brief overview (1-2 sentences).
 
 ## Materials
 
+- **Input data:**
+  - `input table` — what it is, and its conditions: columns must be ..., values should be ...;
+    if they are not, warn that ...
+- **Parameters:**
+  - `theta` — what it controls, its default, and the values it must take.
 - **Software & Repositories:**
   - `ToolName` ([Repository URL]) — Description/version.
 - **Databases & Reference Data:**
@@ -340,7 +362,7 @@ else. The default is to show the header in the result shape.
 **`exact`**: every entry in `expected_outputs` must be present in the result and match.
 `expected_outputs` must have at least one entry; a test that checks nothing is invalid.
 
-**`diagnostic`**: the input violates a requirement the protocol states, and the test checks what the
+**`diagnostic`**: the input fails a condition in Materials, and the test checks what the
 executor does about it. `evaluation.accepted_outcomes` lists the outcomes that count as correct.
 The top-level `expected_outputs` is omitted; each outcome carries its own evidence instead, at
 least one of:
@@ -350,9 +372,12 @@ least one of:
     an executor that refused did not produce the protocol's outputs.
 
 An outcome with neither, or with both empty, is invalid, because a label alone is the executor's
-word. The keys
-`outcome` and `observation` are reserved for the executor's report and may appear in neither
-`expected_outputs` nor `expected_absent`; the validator rejects a test that uses them.
+word. An outcome may add `expected_warning`, a phrase that must appear, case-insensitively, in
+one of the executor's `warnings`. This is how a **should** condition in Materials is checked: the
+executor proceeded, produced the outputs, and said what was wrong with the input. A warning is
+also the executor's word, so it accompanies evidence rather than replacing it. The keys
+`outcome`, `observation`, and `warnings` are reserved for the executor's report and may appear in
+neither `expected_outputs` nor `expected_absent`; the validator rejects a test that uses them.
 
 ```yaml
 evaluation:
@@ -362,6 +387,7 @@ evaluation:
       expected_outputs:
         removed_features: [A, B]
         retained_features: [C, D, E]
+      expected_warning: orientation
     - outcome: refused_for_orientation
       expected_absent: [removed_features, retained_features, retained_matrix]
 ```
@@ -396,6 +422,9 @@ both. The executor does not read `expected_outputs` itself.
 *   In `diagnostic` mode, `result.yaml` has `outcome`, one of the accepted outcomes, and
     `observation`, one sentence saying what was found. If that outcome carries `expected_outputs`,
     their keys follow.
+*   In either mode, `result.yaml` may have `warnings`, a list of strings, one per warning the
+    executor gave because a **should** condition in Materials was not met. It is required when an
+    outcome carries `expected_warning`.
 
 `scripts/check_test_result.py <tests directory> <results directory>` compares every test that has a
 result subdirectory, or one with `--test <name>`, and is the only thing that decides pass or fail. A
@@ -440,10 +469,10 @@ executor:
   language: R                   # what the execution was written in, if known
   invocation:                   # how the protocol reached the agent
     kind: skill                 # skill | in-context | unguided
-    skill: test-protocol 0.1.0  # name and version, for kind: skill
+    skill: test-protocol 0.2.0  # name and version, for kind: skill
   prompt: |                     # verbatim text that started the run
     Run the tests for independent-filtering-variance.
-checker: check_test_result.py 0.1.0
+checker: check_test_result.py 0.2.0
 tests:
   - name: variance_rows
     result: pass                # pass | fail | not_run (no result subdirectory)

@@ -0,0 +1,90 @@
+# 0017. Conditions on Materials: Where They Live and What a Violation Means
+
+- **Status:** Accepted
+- **Date:** 2026-10-23
+- **Deciders:** Levi Waldron
+
+## Context and Problem Statement
+
+Reviewing a Shannon diversity protocol (waldronlab/agent-protocols#53), the rule about rarefaction
+appeared in three places: a Materials bullet, a sentence in Step 2, and a Notes paragraph. None of
+them said what an executor should do with a count table that had not been rarefied. The author had
+no rule to follow: the standard required a `## Materials` heading and recommended two bullet
+groups, software and reference data, and said nothing about where conditions on inputs belong.
+
+Existing protocols had converged on a convention the standard never stated: conditions written as
+prose under Materials, then a "Step 1: Validate" restating them. Steps could contain anything.
+Diagnostic tests referred to "a requirement the protocol states" without saying where, and could
+check only a refusal, through `expected_absent`. There was no way for a protocol to say "proceed,
+but warn", which is what the rarefaction case needs: the index is computable, and the user needs to
+know the values are not comparable across samples.
+
+Issue #46 records the discussion.
+
+## Decision
+
+**Materials is the single home for conditions on inputs.** It lists every input the protocol
+consumes (data, parameters, software, reference data) and every condition an input must or should
+satisfy. Steps consume materials and do not introduce new conditions on them. Statements about what
+the protocol does or does not do belong in the overview or Notes.
+
+**Two severity tiers, written with RFC 2119 words.** A condition written with *must* is a
+requirement: an executor that finds it violated stops, reports which condition failed, and produces
+none of the protocol's outputs. A condition written with *should* is an advisory: the executor
+proceeds, warns the user naming the condition and its consequence, and records the warning with the
+execution.
+
+**The check happens before Step 1.** Every condition in Materials is checked against the inputs
+before the first step runs, so a protocol no longer needs a validation step for anything Materials
+already states. A check that can only be made during computation, such as a sample whose total is
+zero, stays in the step where it arises.
+
+**Scope.** The RFC 2119 reading applies to conditions written in a protocol's Materials section.
+The standard's own prose, which already uses *must* and *should* in their ordinary sense throughout,
+is not reinterpreted.
+
+**The test contract gains the warning tier.** A diagnostic outcome may carry `expected_warning`, a
+phrase that must appear in the executor's `warnings`. It accompanies evidence (`expected_outputs`
+or `expected_absent`) rather than replacing it, since a warning is the executor's word as much as
+the outcome label is. `result.yaml` may carry `warnings`, a list of strings, in either mode;
+`warnings` joins `outcome` and `observation` as a reserved key. The diagnostic result shape names
+`warnings` and withholds the phrase, so an executor warns because the protocol said to, not
+because the shape did.
+
+The protocol-runner skill checks Materials before Step 1 and lists warnings with the departures.
+The test-protocol skill writes them into `result.yaml`.
+
+## Alternatives Considered
+
+- **One tier: every condition is a requirement.** Rejected. In the rarefaction case, the computation
+  is well defined on the input, and the problem is with how the user may interpret the
+  output. Refusing would push authors to write the condition as a note, which is where it was.
+- **A required "Step 1: Validate".** Rejected. It restates Materials, drifts from it, and puts the
+  answer to "what are the inputs" in two places. Existing protocols that have one remain valid; the
+  step is now redundant rather than wrong.
+- **A structured schema for materials** (one YAML block per input with typed constraints). Rejected
+  for now. Conditions like "samples should share a sequencing depth" are prose, and a schema that
+  cannot express them would be worked around. Prose with two reserved words is checkable by an
+  agent and reviewable by a person. A schema waits for a validator that would use it.
+- **Adopt RFC 2119 for the standard document itself.** Rejected. It would give about twenty existing
+  uses of *must* and *should* a meaning nobody intended, each of which would need auditing, for no
+  benefit to protocol authors.
+- **Make `expected_warning` evidence on its own.** Rejected. An outcome checked only by a warning
+  string is checked only by what the executor said about itself, which is the rule ADR 0016 set
+  against labels.
+
+## Consequences
+
+Protocol authors have one place to write conditions and two words to write them with. Reviewers
+have a rule to apply when a condition appears in a step or a note. Executors have a defined action
+for each tier, and the test contract can check both.
+
+The validator is unchanged: it cannot check prose semantics, and the rule is enforceable in review.
+The `expected_warning` and `warnings` keys are validated and checked by the existing tooling.
+
+Existing protocols with a validation step remain valid. Moving their conditions into Materials is a
+content change for each repository, made as protocols are next revised.
+
+The runner's before-Step-1 check is part of its execution contract, which issue #15 proposes to
+move into `PROTOCOL_STANDARD.md`; this ADR puts the condition rules there directly and leaves the
+rest of #15 open.
@@ -20,5 +20,6 @@ This directory contains Architecture Decision Records (ADRs) for the `agent-prot
 - [0014. Ask Two Citation Questions, and Require the Answerable One](0014-two-citation-questions.md) — amends 0009 and 0012
 - [0015. Migrate Protocol Validator to Python and Pydantic](0015-migrate-validator-to-python.md)
 - [0016. Protocol Tests: Fixtures, a Result Contract, and a Review Status](0016-protocol-tests.md)
+- [0017. Conditions on Materials: Where They Live and What a Violation Means](0017-conditions-on-materials.md) — proposed; extends 0012 and 0016
 
 
@@ -35,7 +35,7 @@
 from repo_utils import parse_repository_url  # noqa: E402
 
 CHECKER_NAME = "check_test_result.py"
-CHECKER_VERSION = "0.1.0"
+CHECKER_VERSION = "0.2.0"
 RUN_RECORD_VERSION = 1
 SOURCE_FILE = "source.yaml"
 EXECUTOR_FILE = "executor.yaml"
@@ -233,7 +233,8 @@ def shape_of(case: ProtocolTest, test_dir: Path, results_dir: Path) -> dict:
             if o.expected_absent:
                 entry["expected_absent"] = list(o.expected_absent)
             outcomes.append(entry)
-        shape["result"] = {"outcome": "one of the labels below", "observation": "string"}
+        shape["result"] = {"outcome": "one of the labels below", "observation": "string",
+                           "warnings": "list of strings, when the protocol said to warn"}
         shape["accepted_outcomes"] = outcomes
     return shape
 
@@ -454,10 +455,12 @@ def check_one(case: ProtocolTest, test_dir: Path, result_dir: Path, results_dir:
         return entry
 
     tol = case.evaluation.numeric_tolerance
+    warnings = result.get("warnings")
+    warnings_ok = warnings is None or (isinstance(warnings, list) and all(isinstance(w, str) for w in warnings))
+    failures = [] if warnings_ok else ["warnings: must be a list of strings"]
     if case.evaluation.mode == "exact":
-        failures = compare_top(case.expected_outputs, result, tol, test_dir, result_dir, results_dir)
+        failures += compare_top(case.expected_outputs, result, tol, test_dir, result_dir, results_dir)
     else:
-        failures = []
         label = result.get("outcome")
         outcome = case.outcome(label) if isinstance(label, str) else None
         if outcome is None:
@@ -472,6 +475,12 @@ def check_one(case: ProtocolTest, test_dir: Path, result_dir: Path, results_dir:
             for k in outcome.expected_absent or []:
                 if k in result:
                     failures.append(f"{k}: present in result.yaml, but outcome '{label}' requires it absent")
+            if outcome.expected_warning:
+                needle = outcome.expected_warning.casefold()
+                if not warnings_ok or not any(needle in w.casefold() for w in warnings or []):
+                    failures.append(
+                        f"warnings: no entry contains {outcome.expected_warning!r}, which outcome '{label}' requires"
+                    )
     entry["result"] = "pass" if not failures else "fail"
     if failures:
         entry["failures"] = failures

@@ -153,8 +153,9 @@ def validate_type_and_deps(self):
 
 TEST_NAME_PATTERN = r"^[a-z0-9]+([-_][a-z0-9]+)*$"
 SHA256_PATTERN = r"^[0-9a-f]{64}$"
-# Keys the executor's own report uses in diagnostic mode; a test may not expect them.
-RESERVED_RESULT_KEYS = ("outcome", "observation")
+# Keys the executor's own report uses; a test may not expect them. `outcome` and `observation`
+# belong to diagnostic mode, `warnings` to either mode.
+RESERVED_RESULT_KEYS = ("outcome", "observation", "warnings")
 # Keys the checker writes into run.yaml itself; executor.yaml may not supply them.
 RESERVED_RUN_RECORD_KEYS = (
     "run_record", "protocol", "version", "repository", "protocol_commit",
@@ -306,17 +307,23 @@ class AcceptedOutcome(BaseModel):
     outcome: str
     expected_outputs: Optional[Dict[str, Any]] = None
     expected_absent: Optional[List[str]] = None
+    expected_warning: Optional[str] = None
 
     @model_validator(mode='after')
     def has_evidence(self):
         if not re.match(TEST_NAME_PATTERN, self.outcome):
             raise ValueError(f"outcome label must be lower-case words joined by - or _, found '{self.outcome}'")
         has_outputs = bool(self.expected_outputs)
         has_absent = bool(self.expected_absent)
+        # A warning is the executor's word, like the label; it is checked alongside evidence, not instead of it.
         if not has_outputs and not has_absent:
             raise ValueError(
                 f"outcome '{self.outcome}' needs evidence: a non-empty 'expected_outputs' or 'expected_absent'"
             )
+        if self.expected_warning is not None:
+            self.expected_warning = self.expected_warning.strip()
+            if not self.expected_warning:
+                raise ValueError(f"outcome '{self.outcome}' 'expected_warning' must not be blank")
         if has_outputs:
             _check_no_reserved(self.expected_outputs, f"outcome '{self.outcome}' expected_outputs")
             _check_file_refs(self.expected_outputs, f"outcome '{self.outcome}' expected_outputs", expected=True)

@@ -1,7 +1,7 @@
 ---
 name: protocol-runner
 description: Search, retrieve, evaluate trust, and execute citable workflows from federated protocol repositories
-version: 2.0.0
+version: 2.1.0
 category: protocols
 author: waldronlab
 tags: [workflow, protocol, pipeline, method_origin_citation, provenance]
@@ -96,15 +96,19 @@ silent substitution this skill is meant to prevent:
 
 ### 6. Execute Protocol
 
-1. Follow the steps in the fetched protocol content in order.
-2. **Resource Discovery**: Agents can discover and download pre-computed reference data and upstream tools using the `database_urls` and `upstream_repositories` YAML fields provided in the index.
-3. Adapt the provided code to the user's specific environment, file paths, parameters, and organisms as necessary.
-4. If a step cannot be followed exactly as written, or requires a different package version than specified, note this departure inline.
+1. **Check the materials before Step 1.** Every condition in `## Materials` is written with *must* or *should* (PROTOCOL_STANDARD.md, "Markdown Content Structure"). Check each against the inputs the user supplied:
+   - A *must* condition that fails: stop, report which condition failed and on which input, and produce none of the protocol's outputs.
+   - A *should* condition that fails: warn the user, naming the condition and the consequence the protocol gives for it, then continue. Record the warning for step 7.
+2. Follow the steps in the fetched protocol content in order.
+3. **Resource Discovery**: Agents can discover and download pre-computed reference data and upstream tools using the `database_urls` and `upstream_repositories` YAML fields provided in the index.
+4. Adapt the provided code to the user's specific environment, file paths, parameters, and organisms as necessary.
+5. If a step cannot be followed exactly as written, or requires a different package version than specified, note this departure inline.
 
 ### 7. Record Departures
 
 1. After execution completes, emit a final "Departures from protocol" section.
 2. List any deviations made during execution (e.g., using a different parameter value, skipping a step, or substituting a package). This is a normal part of adapting a protocol; recording it is what matters for provenance.
+3. List every warning given in step 6.1, with the condition that was not met.
 
 ### 8. Generate Draft Methods Section