OPERATIONS.md

Operations

The research harness runs locally with fixed seeds and no production credentials. Every run records source commit, configuration digest, dependency lock digest, authority envelopes, decision certificate, timing, and outcome.

For the supported evaluation entrypoint, copy evaluation/config.example.json, create its output directory, and run telosieve evaluate CONFIG.json. Keep configuration and evidence directories operator-writable but not broadly readable. Treat status 1 as an incomplete evaluation: preserve the directory and inspect possible ledger or temporary certificate evidence before retrying. Use one writer per certificate/ledger path and bind retained reports to their certificate digest. See EVALUATION_CLI.

Install, back up, upgrade, roll back, and uninstall a private evaluation tree only through scripts/evaluation-lifecycle.py. Stop evaluation writers before backup or lifecycle changes. Rollback changes the binary/configuration pair but never rewinds evidence; uninstall requires the canonical root twice and preserves evidence. See EVALUATION_LIFECYCLE.

Generate assessor diagnostics only with scripts/evaluation-diagnostics.py. Treat even its content-redacted digest/size manifest as sensitive, transfer it through an approved encrypted channel, and never substitute it for raw evidence when raw evidence is explicitly required. See OPERATOR_DIAGNOSTICS.

Operational alerts cover signature failure, equivocation, schema mismatch, hypothesis-budget exhaustion, checker disagreement, ledger write failure, and actuator partial failure. No automatic retry may broaden authority or weaken an invariant. Recovery is replay from the last complete ledger record.

Secrets are test keys stored outside fixtures. Logs must not include application values. Retention and deletion policy must be defined before real traces are used.

For OpenTofu v6 evaluation, configure each scripts/opentofu-observation-producer.py instance with canonical absolute OpenTofu, saved-plan, Telosieve, and owner-only key paths. Run producers under separate least-privilege accounts or hosts when those boundaries are available; protect saved plans because they may contain sensitive provider data. A timeout, renderer error, unsafe path/key, malformed output, invalid signature, or byte disagreement is a refusal: preserve diagnostics, do not strip or normalize JSON, and do not retry with fewer producers. The envelope contains the complete plan; do not invoke the producer from a terminal or supervisor that retains stdout. Separate process labels on one host do not establish independent provider, state, backend, clock, or key custody.

On Linux, use the producer isolation kit when producer keys or platform credentials must be separated from the evaluator identity. Create one systemd user and one evaluator-client group per producer; never reuse tokens, keys, credentials, writable directories, or groups across producer fault domains. Treat relay restart loops, authentication refusal, stale sockets, or missing output as observation unavailability rather than permission to fall back to direct execution.

For apply-local, durable service commit occurs before certificate and ledger persistence. An evidence-output failure is therefore an applied-but-unreported incident: do not retry the stale scenario. Preserve the actuator file, lock or temporary artifacts, and output errors; read current state and the durable last-actuation operation digest through local-show; obtain a new authenticated phenotype; and only then evaluate a new operation. Never break an actuator lock until the writer is proven absent.

Local actuator recovery runbook

1. Stop new actuator commands and prove the previous writer process is absent.
2. Preserve the primary state, .witness.json, lock directory, temporary files, certificate, ledger, and stderr for incident analysis.
3. Remove only the stale .actuator.lock directory.
4. Run local-recover; it validates canonical state and witness before removing known temporary files or resolving a pending generation.
5. Run local-show and compare its generation, operation digest, and state digest with the last complete evidence record.
6. If primary state is missing or corrupt but the witness is committed, restore only an exact-latest verified backup with local-restore.
7. Obtain a new authenticated phenotype before further evaluation.

Never restore an older backup, replace the witness, infer a generation, or retry an applied stale scenario. Loss or rollback of both state and witness is a stop condition outside this protocol.

Key lifecycle handling

Keep each lifecycle recovery private key offline and separate from its operational issuer key. Review the exact subject, issuer, authority kind, sequence, parent digest, effective time, and activated/revoked key before signing. Distribute the complete chain, its public recovery root, and an independently obtained trusted tip together. Never repair a missing statement by editing sequence or parent fields, reuse an old operational key, or fall back to a revoked bootstrap key. Missing or conflicting lifecycle state is a refusal and incident condition.

For every enrolled lifecycle, obtain the evaluation time and its bounded trusted-time assertion through the approved operational source. Refuse a window wider than 300 seconds or an evaluation outside its inclusive bounds. The local source label is evidence metadata, not clock authentication; never claim that Telosieve itself proves wall-clock time.

Run ./scripts/run-recovery-ceremony.sh after ceremony-protocol changes. A veto, duplicate, stale, divergent, excluded, or missing-quorum vote aborts the ceremony. Excluding a suspected custodian requires a new request and must not lower the original quorum. Preserve the generated aggregate, but never interpret it as a real recovery-root authorization, signature, custody record, or HSM operation. See RECOVERY_ROOT_CEREMONY.

Before consuming retained certificates, parse them through the supported compatibility boundary and preserve the original bytes. Accept only v7 without an execution extension, v8 with actuation only, or v9 with shadow evidence only. Never relabel a version, strip an unknown field, or invent a newly required trust value. Migration requires verification of the authoritative scenario followed by new evidence generation; the historical certificate remains immutable. See PROTOCOL_COMPATIBILITY.

Use scripts/certificate-reader.py only through bounded standard input and preserve its SHA-256 acceptance summary with the source certificate. A refusal is an incident or migration stop; do not retry after stripping fields or relabelling versions. Run scripts/run-reader-qualification.sh after changing certificate schemas, readers, or version support.

Do not remove a certificate version operationally until its documented deprecation has remained in force for two completed milestones, every registered consumer is inactive or migrated, historical reading remains available, and the separate explicitly approved major-compatibility gate has merged.

When authenticity is required, preserve the certificate and its detached attestation as separate immutable files. Verify both against an independently obtained context, evaluation time, and bounded signer trust set; do not accept an unsigned fallback after requesting verification. Keep retired public keys while their historical attestations must verify, but never extend their signing windows to repair a failure.

Run scripts/run-attestation-qualification.sh after changing certificate, attestation, key-window, or reader logic. The repository’s signing keys are fixtures only and must never be reused for operational evidence.

When timestamp/revocation verification is required, supply the complete bounded timestamp chain, current revocation snapshot, and independently retained exact tips. Treat missing inputs, stale snapshots, tip mismatch, or witness unavailability as refusal; never extend times or roll tips back to restore availability. Run scripts/run-witness-qualification.sh after related changes.

Persist timestamp and revocation tips together with WitnessTipStore; preserve its primary and recovery-witness files as one operational unit. Restore only a create-new backup accepted against the surviving committed witness. On missing, corrupt, locked, pending-inconsistent, stale, or conflicting state, stop verification and preserve all files. Run scripts/run-witness-durability.sh after store or distribution-policy changes.

Run scripts/run-witness-endpoint-harness.sh after endpoint, authentication, transport, retry, timeout, or response-bound changes. Treat every transport failure as evidence unavailability; do not cache past expiry, retry beyond the bound, or accept HTTP success without exact cryptographic verification. Harness tokens and loopback HTTP are fixtures and must not be deployed.

For Post-M25 handoff work, distribute only the frozen request and exact private source through an approved read-only channel. Do not provide endpoint credentials or operate the assessor's environment. Validate returned records with python3 scripts/witness_operator_record.py RECORD.json, then separately verify operator identity, signature, custody, and raw evidence before changing any independent-reproduction checklist state.

For shadow-kubernetes, export the ConfigMap and StatefulSet evidence through a separate read-only process, preserve the raw export, and run Telosieve offline. Treat drift, incomplete observations, stale generations, identity changes, or authority mismatch as an incident/refusal. Never use the shadow certificate as a mutation command; it records a research decision only.

Run ./scripts/run-incident-drills.sh after recovery-protocol changes. A missing or failed drill blocks the milestone; never edit the aggregate result to convert a failure into a pass. See INCIDENT_DRILLS.

For private bundle signing, keep the 32-byte lowercase-hex seed outside the repository and bundle in an owner-only, single-link regular file. Derive and distribute the public trust record through an independent channel. Do not repair verification failures by changing evaluation time, extending windows, replacing trust, or resigning unreviewed bytes. The repository tests only ephemeral keys; an approved operational custody procedure is required before candidate signing.

For HTTP/mTLS PKI monitoring, configure Node Exporter's textfile collector to read the owner-only /var/lib/telosieve/prometheus directory. Alert unless telosieve_http_json_pki_source_valid, telosieve_http_json_pki_fresh and telosieve_http_json_pki_ready are all one and the checked-at timestamp remains within the configured freshness/future-skew window. This timestamp comparison also detects a stopped publisher that leaves an old green textfile behind. Preserve the status and metric files on alert, then inspect certificate lifecycle state offline. Do not silence an invalid or stale source by alerting only on the ready gauge. Independent scraping, routing, retention and response ownership remain deployment gates.

Before proposing a candidate ceremony, inspect evaluation/capabilities.json, evaluation/candidate-profile.json, and evaluation/candidate-test-plan.json inside bundle v3. Require exact agreement with the packaged contract and three configuration files. Run every test-plan command from a source checkout at the manifest's full commit. Any missing mode, digest mismatch, mutation declaration, or non-clean exact-head CI result aborts the proposal; do not edit the ZIP.

Candidate freezing and signing must follow CANDIDATE_SIGNING. Preserve the exact commit and all three artifact digests; source-commit syntax validation alone is not build provenance and must not replace the two-custodian correspondence check.