Repository navigation
Borrowed rig: #516 rig-side max_temp_c edit never reaches the dashboard feed since 2026-09-26 (jobs 1273, 1274) #539
Description
From p2pool-starter-stack/pithead#2741.
The pithead tier4-e2e row rigforge-control: rig-side edit reflected in the dashboard's enriched feed (#516) passed on jobs 1189, 1196, 1201 and 1202 (all 2026-09-25) and failed on the next two jobs that ran it, on unrelated heads, about 17 minutes apart:
1273@94a6e1131131054cb44d7b60c88af346f17167b4(failure)1274@7e45cc28d94e04e40e5b86f52caabd5c88349e96(error, baseline restore; the same row failed before that)
Neither head changes the rig-to-dashboard feed path. On the rig side, _watchdog_json re-reads config.json every 15 s refresh. The dashboard polled the rig every ~30 s through the whole 90 s window with no probe warning in the job's logs.txt. The direct /apply reported applied. So the rig either kept serving max_temp_c=100 or never wrote 102. In job 1273's api-state.json artifact the rig reports RigForge 1.17.4 (an unreleased develop build; v1.17.3 is the latest release), status offline, uptime 1 s, 0 H/s.
Expected: the borrowed rig serves its current config.json in its enriched feed within 15 s of an applied control change, as it did on 2026-09-25. Please check the rig's rigforge-api-refresh.timer, the age of generated_at in its /1/summary, and which RigForge build it runs.
pithead PR for p2pool-starter-stack/pithead#2741 makes the row print the feed's last status and stats rows on timeout, so the next run shows which of these it is.
Moved from p2pool-starter-stack/bench-ci#760 by the runner's triage: The failure concerns the borrowed rig’s RigForge service serving updated config in its feed, not the bench runner or timing; the unreleased RigForge build and rig API refresh need investigation. (recreated: GitHub cannot transfer out of a private repo).
Activity
pithead-developer commentedon Sep 29, 2026
Blocked on p2pool-starter-stack/pithead#2741 and its diagnostic PR #2743. The trusted investigations of jobs 1693 and 1759 found an applied rig-side temperature edit followed by API refresh passes of roughly 65 seconds, but the retained feed artifact was captured after reversal. The current PR captures only the dashboard's last observed value; its review has returned it for the missing direct
/1/summarygenerated_atandrigforge.watchdog.max_temp_cat the failed assertion, before restoration.I read the RigForge refresh path and confirmed that it computes the payload in one timer service pass before publishing the response files. The health block reads the miner's MSR log and memory state on every pass, which are plausible latency points, but there are no internal step timings in the available evidence; assigning a root cause or changing these probes now would be speculative. No code or hardware was changed for this issue.
The Pithead PR owner needs to add the direct-summary capture and its selftest, push it, and run one narrow
tier4-e2erigforge-controljob on that full head SHA. Post its failed-time producer and dashboard values and the job evidence to #2741 and p2pool-starter-stack/bench-ci#760. This issue can resume when that evidence distinguishes a stale RigForge payload from a dashboard feed fault; then the RigForge refresh bottleneck can be fixed and tested on a pushed commit if it is the producer.pithead-developer commentedon Sep 29, 2026
Blocked on p2pool-starter-stack/pithead#2741. Issue #539 is blocked on pithead#2741. Jobs 1693 and 1759 show roughly 65-second RigForge API refresh passes after an applied edit, but the retained feed artifact was captured after reversal, so it cannot establish whether the rig served the old value. I documented the blocker on #539; pithead PR #2743 must capture the direct rig summary alongside the dashboard value at failure and run the narrow rigforge-control job before a RigForge fix can be justified.
The runner resumes this issue when that one closes.
pithead-developer commentedon Sep 29, 2026
Release coordination correction: this issue was recreated from bench-ci#760 while its investigation was active, so the copied body omits the latest timeline and dependency. Preserve your active session for independent source investigation: identify what can hold API refresh passes for roughly 65 seconds, with safe read-only evidence. Root cause/product ownership is not yet proven. The canonical bench investigation #760 is reopened and remains blocked on pithead#2741 / diagnostic PR #2743 for direct rig generated_at/watchdog.max_temp_c and dashboard last-observed values at failure before restore. That Pithead worker is active. Coordinate findings there and avoid duplicate test jobs. Do not claim the transfer or closure resolved the release blocker.
pithead-developer commentedon Sep 29, 2026
The worker has now finished and recorded the same diagnostic dependency as bench-ci#760, with no separate code change or proven RigForge defect. Closing this premature recreation as a duplicate; its source-inspection finding has been preserved on bench-ci#760. Continue the single coordinated reproduction there after pithead#2741 / PR #2743 provides the required capture.
pithead-developer commentedon Sep 29, 2026
Reopened with failed-time producer evidence
Pithead's diagnostic head reproduced the #516 row in job 1825. At the failed assertion, its direct authenticated rig
/1/summarystill reportedgenerated_at=2026-09-29T20:48:51Zandrigforge.watchdog.max_temp_c=100after the control edit to 102 had been applied. The dashboard's last observed row wasstatus=offline, stats: Agent report=stale for 1m 32s. The producer's own stale body explains this occurrence; a dashboard feed repair cannot make the rig publish the newer value.The API refresh service was active, but read-only service history again showed passes lasting about 65 seconds during the assertion window. The retained logs do not time the operations inside a pass. RigForge needs to identify the slow operation in
api_refresh/_api_rigforge_block, fix the latency at that shared cause, and verify the 15-second refresh contract under mining load. Preserve the 90-second assertion and the read-only API boundary. Bench-ci#760 remains the canonical cross-repo investigation and is blocked here until the producer fix is tested; Pithead #2741 / PR #2743 supplies the failed-time diagnostic.bench-ci job 1825@2bdd01a69280f1de449f47cd3a0fecc16a5b862d · pithead · tier4-e2e
Generated by
bench-ci evidence 1825@2bdd01a69280f1de449f47cd3a0fecc16a5b862d; re-run it to check.pithead-developer commentedon Sep 29, 2026
The producer-side finding from pithead#2743's new failure detail (the rig's own feed stops refreshing after a direct max_temp_c control-apply, seen on two jobs) is owned by rigforge#540. This issue (moved here from bench-ci#760) keeps the bench-side trail; close it as a duplicate of #540 or keep it for the rig's timer, at the maintainers' choice.
pithead-developer commentedon Sep 30, 2026
The producer defect and focused fix are owned by rigforge#540 / PR #541. This recreated issue has no separate implementation or active worker and duplicates the same stale enriched-feed failure. Closing it as superseded so it cannot indefinitely hold bench-ci#760. This is duplicate cleanup, not a passing test result: rigforge#540 must still complete its bench-ci#952 prerequisite, exact-head control proof and review before the producer bug is resolved.