[pithead-developer]

Closes #569

Related unresolved failure: #563.

The nightly network phase reports a combined metadata/key-superset failure without retaining which condition failed. This draft adds fixed boolean network-summary observations from the same responses, omitting bodies, key names, timestamps and parser errors. The original strict assertion, failure row and configuration lifecycle remain intact. The required regression and test documentation cover the diagnostic behavior.

The isolated network reproduction passed, so it does not establish the nightly root cause. A new narrow network-sequence phase reproduces the nightly connect → worker-api → api-impact → network prefix in one process and snapshot, stopping after network. Separate phase submissions restore between invocations and cannot preserve that state. Fatal failures and failed rows stop the sequence through the existing cleanup path. This targets the original failing sequence without running unrelated later phases.

Decisions taken

• The operator authorized a scope split: this existing PR closes diagnostics prerequisite #569 on maintenance & gates. The unresolved failure #563 remains open. The later owner ruling of 2026-10-03 moved it to maintenance & gates as a soak watch; a recurrence in the gated run or soak returns it to the release scope. No second implementation or PR was created.
• Passing reproductions validate the instrumentation and narrow reproduction phase. They establish neither a root cause nor a corrective fix for bench-ci nightly: tier4-e2e failure on develop@a625d1d #563. The strict assertion and failure row in tests/e2e-pithead.sh remain intact.
• Current head 48b227098a8eae72503de741de18d12d740fe6b3 is unchanged. Completed job 2054@48b227098a8eae72503de741de18d12d740fe6b3 supplies current-head tier4-e2e proof; any head change requires fresh evidence.
• This diagnostic prerequisite is submitted to the separate adversarial-review gate. Author-side verifier/security reviews do not replace that gate. The PR stays draft for the reviewer and runner; the author does not merge it.
• After this prerequisite lands, the next scheduled rigforge nightly supplies recurrence evidence for bench-ci nightly: tier4-e2e failure on develop@a625d1d #563. Any recurrence must be diagnosed from its full logs and artifacts; no speculative rerun loop is authorized and bench-ci nightly: tier4-e2e failure on develop@a625d1d #563 must not close as fixed from these passing runs.

Ponytail review: retained bounded diagnostics, the minimal same-process reproduction phase, mandatory regressions and corrected documentation; no retry, relaxed assertion, product change or new dependency.

Validation at 48b227098a8eae72503de741de18d12d740fe6b3:

• timeout 60 bash "$TMPDIR/rigforge-563-summary-unit.sh" — PASS, 27 assertions. The scratch driver uses repository assertion helpers and sources tests/test-summary-diagnostics.sh; no hardware used.
• timeout 60 bash "$TMPDIR/rigforge-563-gates-unit.sh" — PASS, 62 assertions. The same driver sources the existing tests/test-e2e-gate-fail-closed.sh, including the new dispatch regression.
• Restoring the pre-sequence helper makes the summary driver FAIL with five assertions; restoring the corrected helper passes. The earlier baseline-harness mutation also failed the required phase-diagnostic assertion.
• shellcheck --severity=warning tests/e2e-summary-diagnostics.sh tests/test-summary-diagnostics.sh tests/test-e2e-gate-fail-closed.sh — PASS. The main harness reports baseline SC2120 under local ShellCheck 0.9.0; complete local ShellCheck PASS is not claimed. GitHub uses pinned 0.11.0.
• shfmt -i 4 -d tests/e2e-pithead.sh tests/e2e-summary-diagnostics.sh tests/test-summary-diagnostics.sh tests/test-e2e-gate-fail-closed.sh, FILE_BUDGET_REQUIRE_BASE=1 bash scripts/lint-file-budget.sh, make lint-topology, git diff --cached --check — PASS.
• Local Markdown check unavailable: npx is absent. The earlier head's GitHub Markdown check passed.
• Independent author-side verifier and security reviewer — clean again for the added sequence. Static review is not a hardware result or merge approval.

All eight required GitHub checks passed at current head 48b227098a8eae72503de741de18d12d740fe6b3: suite, Docker e2e, coverage, shell/Markdown/YAML lint, secret scan and workflow audit. Complete suite and container e2e were not run locally.

Job 2052's full status, complete log and sole retained artifact were read. The network metadata/key contract and secret-leak checks passed with verified restoration. Its diagnostic flags show valid metadata, matching keys and no unreachable-miner marker. This proves the isolated phase on that head, not the preceding nightly sequence. That isolated run alone did not establish intermittent behavior because its phase sequence differed from the nightly. The subsequent network-sequence job runs the same prefix through the original assertion, which passed without any product code or assertion change. The related original issue #563 is also labeled flaky; the failure is not attributed to product logic or bench timing without causal evidence.

Endpoint access recovered under bench-ci#1144; its infrastructure cause is tracked separately in bench-ci#1146. The missing submitter guide remains tracked in bench-ci#1028. The queued evidence ID mismatch is tracked in bench-ci#922; both completed reproduction blocks now print consistent submitted IDs. All evidence below was regenerated after completion and retained unedited.

Historical nightly evidence (symptom/restoration only):

bench-ci job 349@a625d1d497978d320a6e1b01125717fdbec667ca · rigforge · tier4-e2e

• options: {"phases": ["all"]}
• status: completed · conclusion: failure · exit 1 · 4m 53s
• baseline verified: yes · rig restored: yes
• node guard: not recorded · no comparison recorded
• steps: checkout success · reserve success · preflight success · run failure · collect success · restore success · cleanup success
• rows: 15 pass · 1 fail · 0 skip
• failed rows:
  • sister summary lacks valid metadata or its XMRig key superset differs (no issue linked)
    Generated by bench-ci evidence 349@a625d1d497978d320a6e1b01125717fdbec667ca; re-run it to check.

Isolated network reproduction:

bench-ci job 2052@7088ca6a402c0910519d81013c799b848709d737 · rigforge · tier4-e2e

• options: {"phases": ["network"]}
• status: completed · conclusion: success · exit 0 · 2m 43s
• baseline verified: yes · rig restored: yes
• node guard: not recorded · no comparison recorded
• steps: checkout success · reserve success · preflight success · run success · collect success · restore success · cleanup success
• rows: 8 pass · 0 fail · 0 skip
  Generated by bench-ci evidence 2052@7088ca6a402c0910519d81013c799b848709d737; re-run it to check.

Nightly-prefix reproduction (no failure reproduced):

bench-ci job 2054@48b227098a8eae72503de741de18d12d740fe6b3 · rigforge · tier4-e2e

• options: {"phases": ["network-sequence"]}
• status: completed · conclusion: success · exit 0 · 6m 16s
• baseline verified: yes · rig restored: yes
• node guard: not recorded · no comparison recorded
• steps: checkout success · reserve success · preflight success · run success · collect success · restore success · cleanup success
• rows: 19 pass · 0 fail · 0 skip
  Generated by bench-ci evidence 2054@48b227098a8eae72503de741de18d12d740fe6b3; re-run it to check.

[pithead-developer]

Blocked on bench-ci#1144: approved CI list and narrow reproduction submission now fail with connection refused. No job id was returned, so no hardware result or passing reproduction is claimed. Bench maintainers must restore endpoint availability; resume the same network submission at 7088ca6a402c0910519d81013c799b848709d737 with idempotency key 9faf250079254db39bd32d5175ff9258 to safely recover any accepted job.

The bounded diagnostic change, required focused regression (22 assertions), documentation and author-side verifier/security reviews are complete in draft PR #568. Full historical job logs and the sole artifact were read, but they lack the observations needed to establish the cause. Root-cause diagnosis and correction remain unfinished. Once a recovered/new diagnostic job completes, read its full logs/artifacts, renew generated evidence and continue on this same PR. No manual bench or rig action was performed.

[pithead-developer]

The operator closed bench-ci#1144 after restoring endpoint availability. The original idempotency key resumed submission as job 2052@7088ca6a402c0910519d81013c799b848709d737, restricted to network. All eight current-head GitHub checks passed. PR #568 remains draft and awaits the diagnostic job; root-cause correction is still outstanding. Generated evidence is in its body; the queued identifier mismatch is reported on existing bench-ci#922 and will be checked again after completion.

[pithead-developer]

The completed isolated network job 2052@7088ca6a402c0910519d81013c799b848709d737 passed; its full status, log and sole artifact were audited. It does not reproduce the preceding nightly phase state. PR #568 now adds a narrow network-sequence reproduction, sharing one process/snapshot across the four-phase nightly prefix and omitting unrelated later phases. Focused gate regressions and author-side verifier/security reviews passed. Job 2054@48b227098a8eae72503de741de18d12d740fe6b3 is queued for that sequence; root-cause correction remains unfinished. Refreshed unedited evidence for every job is in the PR body.

[pithead-developer]

Audited completed job 2054@48b227098a8eae72503de741de18d12d740fe6b3: full status, complete log and sole retained phase artifact. The nightly prefix passed, including the unchanged summary metadata/key assertion, with valid diagnostic flags and verified restoration. All eight required GitHub checks passed at this head; focused regressions and author-side reviews are clean. Every generated evidence block in draft PR #568 has been refreshed unedited.

No product code or strict assertion was changed, and the original cause remains unproved. This issue retains the intermittent failure and is also labeled flaky; no further rerun loop is being started. PR #568 is complete as bounded diagnostic instrumentation and a narrow reproduction phase, but it cannot honestly be claimed to resolve the original failure. It remains draft pending operator disposition of that distinction. No merge-readiness claim is made for closing #563.

[pithead-developer]

Ready for review

This PR owns diagnostics prerequisite #569; #563 remains the unresolved failure. At 48b227098a8eae72503de741de18d12d740fe6b3, it adds bounded boolean summary observations and a narrow same-process nightly-prefix reproduction while retaining the strict assertion, lock/snapshot lifecycle and failure restoration. Mandatory regressions and corrected test documentation accompany the change.

All eight current-head GitHub checks passed. The focused gate driver passed 62 assertions; mutations removing the instrumentation or sequence fail the relevant regressions. Completed job 2054@48b227098a8eae72503de741de18d12d740fe6b3 exercises the new sequence and the original contract assertion, with verified restoration. Its full status/log/sole artifact were audited; unedited generated evidence for all jobs is in the body. Author-side verifier/security reviews are clean, and the file-budget freeze and topology checks pass.

No root cause or corrective fix for #563 is claimed. Later unrelated hardware phases were omitted by design; the completed tier4 run covers the diagnostic scope. Local full-suite/container execution was omitted in favor of the actual GitHub checks, and local Markdown tooling was unavailable; the GitHub Markdown check passed. The code head is unchanged, so its completed proof remains current.

Please route prerequisite #569 / this draft PR to a separate reviewer session at that head, re-derive the evidence, and record the required adversarial-review status. Author-side reviews are not that approval. The runner may land it after that gate passes; #563 remains open for the next scheduled nightly recurrence rather than a rerun loop.

[pithead-developer]

The operator split the tested diagnostic work into #569 on maintenance & gates, reusing PR #568. Its closing reference now targets only #569; #563 remains open on v2 - appliance for the unresolved intermittent failure.

#563 is blocked on #569 landing: the diagnostic code must be available to the scheduled nightly before a recurrence can supply the missing observations. The implementation, mandatory regressions, documentation, author-side reviews, all eight current-head GitHub checks and audited tier4-e2e job 2054@48b227098a8eae72503de741de18d12d740fe6b3 are finished for the prerequisite. PR #568 has been submitted for the separate adversarial-review gate; author-side reviews do not replace it. The independent reviewer must record a passing status at the tested head and the runner must land the prerequisite. That clears this dependency; afterward the next scheduled rigforge nightly is the evidence source, and only a matching failure resumes causal diagnosis. No speculative rerun is being started and no passing reproduction closes #563 as fixed.

[pithead-reviewer]

adversarial-review: PASS at 48b2270, round 1, by the pithead-reviewer reviewer (a fresh session that did not produce this head; gpt-6.1-sol).

PASS at 48b2270 for #569: sanitized boolean diagnostics, preserved strict assertion and restoration, and a tested same-process nightly-prefix reproduction. Current-head GitHub checks and deployed tier4 evidence hold; #563 remains unresolved.

Evidence

• pwd; git status --short; git rev-parse HEAD; cat AGENTS.md: clean worktree at 48b2270; repository rules read.
• Read graphify, ponytail and ponytail-review SKILL.md: applied graph-assisted inspection and complexity review; no unnecessary machinery found.
• gh pr view 568 --repo p2pool-starter-stack/rigforge: read body, all comments, commits and reviews. Base is develop; latest Ready for review is 2026-10-03T07:09:34Z; no earlier RETURN review.
• gh issue view 569 --repo p2pool-starter-stack/rigforge: acceptance requires boolean-only diagnostics, unchanged assertion and lifecycle, ordered shared-state prefix, failure propagation, regressions, corrected documentation and current-head proof.
• gh issue view 563 --repo p2pool-starter-stack/rigforge: read original failure and comments; scope split explicitly leaves this failure open. Direct external references were read from runner-prefetched sources: #1144/#1146 closed; #1028/#922 open.
• git diff --stat origin/develop...HEAD; git diff origin/develop...HEAD; git log --oneline origin/develop..HEAD: five files, two commits, confined to diagnostics, reproduction, regressions and documentation.
• cat CONTRIBUTING.md; inspected Makefile, CI workflow and affected harness/test files: base, file-budget, documentation and test wiring comply. Strict assertion, failure row, lock, snapshot and restoration remain intact.
• graphify --help; graphify extract . --code-only --out --no-cluster: scratch graph built with 817 nodes and 1621 edges; unsupported files and zero-node sources reported explicitly.
• graphify query 'summary_contract_diagnostics phase_network_sequence phase_network' --graph --budget 1200: traced affected helpers and harness relationships.
• gh pr checks 568 --repo p2pool-starter-stack/rigforge: all eight checks pass; none skipped.
• gh api /check-runs: runner wrapper refused the read. No status or check run was written.
• gh run view 37104203622 --repo p2pool-starter-stack/rigforge: exact head, completed success; suite, Docker e2e, coverage and shell/Markdown/YAML lint all successful.
• gh run view 37104203646 --repo p2pool-starter-stack/rigforge: exact head, completed success; secret scan and workflow audit successful.
• source configured bench-ci environment; bench-ci --help; read matching submitter guide: approved evidence client available; no submission or manual bench action performed.
• bench-ci evidence 349@a625d1d497978d320a6e1b01125717fdbec667ca: exact pasted-block match; historical failure, 15 pass/1 fail/0 skip, baseline verified and rig restored. Its summary failure is tracked by bench-ci nightly: tier4-e2e failure on develop@a625d1d #563; not current-head proof.
• bench-ci log 349@a625d1d497978d320a6e1b01125717fdbec667ca; bench-ci status 349@a625d1d497978d320a6e1b01125717fdbec667ca --json: full records read; original summary assertion failed and restoration completed.
• bench-ci evidence 2052@7088ca6a402c0910519d81013c799b848709d737: exact pasted-block match; success, 8 pass/0 fail/0 skip, baseline verified and rig restored. Supports the older isolated reproduction only.
• bench-ci log 2052@7088ca6a402c0910519d81013c799b848709d737; bench-ci status 2052@7088ca6a402c0910519d81013c799b848709d737 --json: full records read; branch checkout, boolean diagnostics, original assertion and secret-leak checks verified.
• bench-ci evidence 2054@48b227098a8eae72503de741de18d12d740fe6b3: exact pasted-block match; current-head success, 19 pass/0 fail/0 skip, baseline verified and rig restored.
• bench-ci log 2054@48b227098a8eae72503de741de18d12d740fe6b3; bench-ci status 2054@48b227098a8eae72503de741de18d12d740fe6b3 --json: full records read; deployed branch ran connect → worker-api → api-impact → network, retained strict assertion, emitted fixed booleans and stopped after network.
• bench-ci artifacts --help; bench-ci artifacts for jobs 349, 2052 and 2054; downloaded and read each sole retained phase log: artifacts agree with job logs and evidence counts.
• git worktree add --detach HEAD; timeout 60 bash : repository assertion helpers exercised test-e2e-gate-fail-closed.sh; 62 pass/0 fail.
• Scratch mutation driver: baseline harness produced 60 pass/2 fail; pre-sequence helper produced 57 pass/5 fail; reverting all new harness behavior produced 41 pass/21 fail. Restored head returned 62 pass/0 fail.
• timeout 60 bash test-summary-diagnostics.sh: 27 pass/0 fail, including sanitization, phase wiring, shared state, sequence order and fatal/failed-row propagation.
• shellcheck --version; shfmt --version; command -v timeout: ShellCheck 0.11.0, local shfmt 3.13.1 and timeout available.

Findings

• [note] rigforge-563-linked.md:28; rigforge-563-linked.md:29: The detector flags two lines in older linked issue bench-ci nightly: tier4-e2e failure on develop@a625d1d #563. Operator follow-up only; PR test: retain network summary diagnostics and prefix reproduction (#569) #568 and issue Retain bounded diagnostics for intermittent network summary failures #569 scan clean.
• [note] make lint: Local shfmt 3.13.1 rejects unchanged baseline formatting in tests/run.sh. Identical base/head hunks were confirmed, and verified CI-pinned shfmt 3.14.1 passed the complete shell-file set.