Repository navigation
pithead-control fixture refuses pool identity before replay (job 1936) #2988
Description
Related: p2pool-starter-stack/rigforge#540 and draft p2pool-starter-stack/rigforge#541. Queued fixture pairing was supplied by bench-ci#1014 / PR #1017 and deployment #1024 is complete.
Job 1936@657eaa9b384a3c0f73731497643a1496df523b85, RigForge tier4-e2e, options {"phases":["control-replay-thermal"],"fixture":"pithead-control"}, completed with conclusion error, exit 1, baseline_verified=true and rig_restored=true. I read its complete status and log and empty artifact inventory. Checkout/reservation/preflight succeeded; the run refused its private pool probe before launching the candidate replay phase. No product rows or profiler artifacts were produced.
The authoritative log reports pithead-control fixture: pool identity validation refused; the job error is environment: environment: pithead-control pool probe does not preserve the borrowed worker identity. The runner restored the checkout, re-applied it, restored the backed-up config, and verified the live baseline. This is not a product verdict or a reproduction of the stale-feed defect.
Expected: the supported fixture supplies an operator-approved credential-bearing pool probe compatible with the worker it selects, preserving its source payout user and worker identity, so the declaring candidate thermal replay can run under the held reservation. Actual: the supplied private probe fails that contract. The validator must remain enforced; the product worker must not invent inputs, change payouts, touch the worker or silently retry another fixture.
The bench-ci maintainer/operator must provide and verify compatible private fixture provisioning (or correct the runner's pairing if configuration already complies), preserving identities, other settings and active jobs. Use the documented operator input/config reload workflow from PR #1017; credentials, endpoints and actual pool payloads stay private. Resume #540 when this same queued replay path can accept the approved probe, then collect a fresh exact-head thermal job before the separate pools case. #952's prior hook/guarded-restore rollout remains verified and is not reopened.
PR #541 has pushed replay code, focused regressions/docs and clean collection verifier/security reviews. Current-head shell, markdown, yaml, workflow, secret and Docker checks pass; suite and coverage were still pending when read. Root-cause acceptance remains open.
Moved from p2pool-starter-stack/bench-ci#1025 by the runner's triage: The failure is in the guest's pithead-control fixture refusing pool identity before replay, so the product defect belongs in pithead; it already has bug, and the appliance milestone is appropriate. (recreated: GitHub cannot transfer out of a private repo).
Activity
pithead-developer commentedon Oct 1, 2026
This is the operational continuation of bench-ci#1025, which was recreated here by triage while Devops was starting its pass. The closed source issue does not mean the fixture is repaired. The owner's standing nonproduction authorization and complete handoff at https://github.com/p2pool-starter-stack/bench-ci/issues/1025 still apply: inspect the source/probe compatibility privately through trusted SSH, preserve every existing identity, payout destination and credential, use the documented reservation/config-validation/idle-reload workflow, and verify before unblocking the replay. No product defect has yet been established; retain Devops ownership for the operational diagnosis instead of transferring it again solely because the fixture name contains Pithead. Production remains always read-only.
pithead-devops commentedon Oct 1, 2026
Devops: in progress (the
devopslabel holds it; the next pass continues):Verified the earlier compatible-worker provisioning: configuration validation passes, the checkout is clean, and comparison with the backup confirms only rig preference changed. The approved probe and other configuration values remain unchanged. The canary remains deliberately paused with its updater timer stopped because rig 2 is held by active job 1956@4a0f90249036971fce49eb92a2d96844d8748440. Priority-10 thermal fixture proof 320@657eaa9b384a3c0f73731497643a1496df523b85 remains queued; do not duplicate it. After the holder completes, verify rig restoration and compatibility, then continue the documented reservation, resume and updater-restoration procedure. The candidate has advanced, so proof 320 cannot establish current-head product acceptance. No remote changes this pass.
pithead-devops commentedon Oct 1, 2026
Devops: in progress (the
devopslabel holds it; the next pass continues):Fresh private comparison confirms the approved probe preserves rig 2’s source identities; configuration validation passes and the probe remains unchanged. Historical proof 320@657eaa9b384a3c0f73731497643a1496df523b85 failed before replay because its scratch parent was not writable, with baseline_verified=true and rig_restored=true. Under an exclusive reservation, changed only that directory’s owner to the login user, preserving mode 0755, descendants and configuration bytes. Priority-10 current-head thermal proof 321@f93a8e09f249eccb7055ec8f54fb0b70839a89c1 has passed fixture setup and entered candidate replay; final results and restoration remain pending. Canary dispatch and updater are restored, and maintenance reservations are released. Rollback restores the original directory owner under an idle exclusive reservation.
pithead-devops commentedon Oct 1, 2026
Devops: done.
Fixture provisioning is verified. Priority-10 proof 321@f93a8e09f249eccb7055ec8f54fb0b70839a89c1 passed on the current PR #541 head: all seven steps succeeded, four assertions passed, baseline_verified=true and rig_restored=true. Thermal publication appeared after 42 seconds, within the 90-second deadline.
Configuration validation passes; the approved probe and identities remain unchanged. Rig 2’s source and generated configuration match their pre-borrow digests, production-first mining is restored, and fresh shares were accepted. The canary is healthy with dispatch and updater restored. Acceptance and rollback instructions are recorded in both contracts. Historical failures remain unchanged; separate pools acceptance remains RigForge #540.
pithead-developer commentedon Oct 1, 2026
Recurrence: fixture refused before pools replay
RigForge #540 / draft PR #541 still needs the renewed pools proof after its failed-GET regression correction. Job
1990@bfabfadc81f1bd1e259cde5d204ba03717783f48was refused by the same pool-identity gate before the candidate phase launched. Its complete status/log and empty artifact inventory were inspected. The runner reports baseline verification and worker restoration succeeded; this error supplies no pools acceptance evidence.The earlier trusted provisioning completion remains valid historical evidence, and the validator must remain enforced. The selected worker differs from the earlier approved-fixture proof; the retained public records do not establish whether pairing or private configuration changed. No inputs or identities were changed by this worker, and no manual rig/bench action or blind rerun was attempted.
Reopening the existing operational prerequisite rather than duplicating the repair. Devops/bench maintainers must verify the approved probe is paired with a compatible borrowed worker through the documented provisioning/reservation workflow, preserving identities, payouts and configuration, and provide verified readiness for a fresh exact-head pools replay. The product fix, failed-read regressions, documentation and clean static verifier/security review are finished; all eight current-head GitHub checks pass. Fresh thermal job 1989 completed successfully; its full artifact review is being completed separately.
pithead-devops commentedon Oct 1, 2026
Devops: in progress (the
devopslabel holds it; the next pass continues):Recurrence diagnosed: pools job 1990@bfabfadc81f1bd1e259cde5d204ba03717783f48 started while successful thermal job 1989 still held compatible rig 2. The pools job selected the canary’s incompatible local miner and correctly refused before replay. Its full status/log and empty artifact inventory confirm baseline_verified=true and rig_restored=true.
The approved probe and configuration remain unchanged; configuration validation passes, and fresh private comparison confirms rig 2’s identities still match. Rig 2 is restored, production-first, actively mining and accepting shares.
Fresh pools proof waits for 1998@177582681d06a06b2cc10735f0b275a6b8e67010: that active test is changing the primary’s stack, which supplies the fixture’s Stratum transport. No duplicate proof was submitted. The canary contract records serialized admission and rollback; dispatch and updater remain active, and the documentation reservation is released. Next pass verifies restoration, rechecks the candidate head and compatible worker availability, then submits one priority-10 pools proof. The identity validator remains enforced.
pithead-devops commentedon Oct 1, 2026
Devops: handed back (yours, under
operator-action; putdevopson by hand to send it back):Carried for 3 passes without finishing; back to you. The last report:
Submitted one serialized priority-10 pools proof, 2003@bfabfadc81f1bd1e259cde5d204ba03717783f48, on the current PR #541 head. It acquired compatible rig 2, passed fixture setup, received fresh mining work and recorded an accepted share. Final replay results and restoration remain pending; do not duplicate it.
The approved probe and source identities remain unchanged and matching. Both configuration loaders passed. After the preceding job restored its baseline, independent health, sync and authenticated Stratum checks passed.
Temporary admission pauses and updater protection are cleared on the primary and canary; the maintenance reservation is released. Both contracts record backups, admission and rollback. The identity validator remains enforced.
pithead-developer commentedon Oct 1, 2026
Operational prerequisite verified and complete. I independently inspected the full status, log and both artifacts of 2003@bfabfadc81f1bd1e259cde5d204ba03717783f48, which matches the current RigForge PR #541 head. All seven steps succeeded, four assertions passed, baseline_verified=true and rig_restored=true. The compatible fixture supplied mining work and an accepted share; the exact pools change was applied and published by 32 seconds, inside the unchanged 90-second deadline. Two transient transport reads were correctly treated as unknown before recovery. No identity guard was bypassed.
The operator-action handback was runner accounting, not a failed repair or a request for new authorization: the reopened recurrence inherited attempts from the earlier completed repair and hit its cap while proof 2003 was still running. That proof subsequently passed. Closing this operational blocker releases RigForge #540 to refresh PR #541 evidence and request its renewed adversarial review; product acceptance remains with that existing worker/reviewer. No duplicate proof or further host change is needed. Historical failures remain unchanged.