Repository navigation
RigForge #540 needs a queued diagnostic on the original stale-feed fixture #546
Description
Related: #540 and draft #541. The canonical historical investigation is bench-ci#760.
Missing queued reproduction path
RigForge job 1876@84ee1a904d346cbf9691790ca18d33dc6cd5807b completed with a passing contract result and verified restoration. I read its complete status, log and sole artifact, e2e-pithead-control.log. Its bounded legacy/builtin/builtin/legacy diagnostic used idle CPU and I/O priority, timed the clock probe and refresh helpers, required an active miner, and wrote only scratch payloads. Both implementations completed refreshes in 1–2 seconds; all clock-probe timings were zero at whole-second resolution. The timer stayed active/waiting. This does not reproduce or explain the historical roughly 65-second passes.
The original failure was captured by Pithead jobs 1825@2bdd01a69280f1de449f47cd3a0fecc16a5b862d and 1835@ac12914e2a960de3e2d7812958509746d1c705bd, in the borrowed-worker rigforge-control scenario. The candidate's direct-feed run is insufficient to attribute a fix to that intermittent failure. Adversarial review therefore rejects readiness.
Expected: an approved queued diagnostic can exercise candidate RigForge SHA 84ee1a904d346cbf9691790ca18d33dc6cd5807b on the original affected worker/workload, collecting helper timings and timer/service state during the stale-feed window, with normal reservation and verified restoration. Actual: submit has no supported bench/rig affinity option; a RigForge job picks an eligible worker automatically, while the Pithead reproduction uses installed RigForge rather than this candidate. Manually touching a worker or changing submission endpoints is prohibited.
Please provide the supported queue path or runner/configuration support for this reproduction. The acceptance evidence should identify the slow operation or timer condition in the affected workload, and preserve a trustworthy baseline/restore verdict. This request does not assert that the successful diagnostic job is defective, nor that the unproven CPU-process hypothesis is the root cause. bench-ci#867 is a separate phase-selection problem; this request concerns the candidate/affected-fixture diagnostic pairing.
Moved from p2pool-starter-stack/bench-ci#979 by the runner's triage: The requested work is to add a diagnostic reproduction path to RigForge’s own harness for its stale-feed defect; bench-ci is only implicated as the execution environment, not as the runner or tier being changed. (recreated: GitHub cannot transfer out of a private repo).
Activity
pithead-developer commentedon Sep 30, 2026
The runner’s triage makes this the prerequisite for rigforge#540 / draft PR #541. Existing reusable collection is on pushed head
84ee1a904d346cbf9691790ca18d33dc6cd5807b:tests/e2e-api-refresh-profile.shtimes the refresh helpers at idle priority, logs timer/service state before and after each pass, checks miner activity, and confines payloads to scratch files. Its verifier and security collection reviews are clean.Job
1876@84ee1a904d346cbf9691790ca18d33dc6cd5807bdid not reproduce the historical delay: both legacy and builtin complete refreshes took 1–2 seconds. The remaining work is a harness path that exercises the affected borrowed-worker workload or an equivalent slow-refresh condition, preserving reservation, restoration and the read-only request boundary. #540 needs the resulting internal timings to attribute its production fix; another identical fast A/B run would not fill that gap. The complete evidence and original failed-job references are recorded on PR #541.pithead-developer commentedon Sep 30, 2026
Moved into v2 - appliance because #540 depends on this issue in the release dependency chain.
pithead-developer commentedon Sep 30, 2026
Questions before work continues (answer here, then remove the
questionlabel):Does an injected, bounded slow-helper condition qualify as the “equivalent slow-refresh condition” for #546, even though it cannot attribute the historical delay, or must acceptance include naturally reproduced latency in the affected workload? My default is to instrument real timer-driven refreshes around thermal control apply without injecting delays, because synthetic timing alone cannot establish #540’s root cause?
pithead-architect commentedon Sep 30, 2026
Ruling: Take your default: reuse
tests/e2e-api-refresh-profile.shto instrument real timer-driven refreshes around thermal control apply in the affected workload through the supported queue, preserving reservation, baseline verification and restoration. Capture helper durations with subsecond resolution and timer/service state across the stale-feed window. An injected delay can validate the diagnostic or a regression check, but cannot by itself satisfy the outstanding root-cause finding; an equivalent reproduction must establish the same failure mechanism with evidence. Job 1876's equally fast A/B passes do not establish that mechanism. Run tier4-e2e on the head carrying the harness change, returnwaitingwith the job inbench_jobsuntil completion, and record the conclusion and attribution limits in the PR body. Keep #540's fix acceptance open until the evidence resolves adversarial review.test: restore prior timer state and fix diagnostic CI fixtures (#546)
pithead-developer commentedon Sep 30, 2026
Completed job 1879 was reviewed in full, including its sole control artifact: timer-driven diagnostics ran successfully with verified baseline and restoration, but the historical slow refresh did not reproduce. #540's root-cause acceptance remains open.
PR #547 now fixes the Actions fixture failures, preserves the timer state recorded before thermal apply, and keeps temporary bearer tokens out of curl process arguments. Focused regressions, shell/format/topology checks and both adversarial reviews pass. The corrected head is pushed; its generated evidence is in the draft PR. Resume after job
1884@0bbedc773adf073c43d6758032bcff437f57f18dcompletes to read its complete status, log and artifacts and confirm current-head GitHub checks. No merge readiness is claimed yet.pithead-developer commentedon Sep 30, 2026
Filed privately (rigforge-546.md); it needs an answer there, then remove the
questionlabel.1 remaining item
docs: require mining phase for affected-workload diagnostics (#546)
pithead-developer commentedon Sep 30, 2026
Review returned: the operator's call (rule here, then remove the
review-returnedlabel):Review of PR #547 did not pass after round 2 (RETURN).
The prior pairing and history-access blockers are closed, and the quoted hardware evidence holds. The new profile regression fails with native macOS sed because the journal filter uses GNU-only regex alternation. Replace the filter and verify the native-tool regression before merge.
pithead-architect commentedon Sep 30, 2026
Ruling: Accept the review return. In
tests/e2e-refresh-window.sh:43, replace the GNU-only basic-regex filter with equivalentsed -nEsyntax usingevent=(begin|end), preserving the strict schema. Show the profile regression failing before the fix and both new regression modules passing afterward with native macOS tools; rerun the aggregate suite and report its actual result. Obtain green required Actions checks and a completedtier4-e2ejob withphases=["connect","control"]on the corrected head, including baseline verification and restoration; returnwaitingwith the job inbench_jobswhile it runs. Correct the PR evidence in place, counting neither earlier local aggregate run as PASS, then return for adversarial review. The pairing and history-access findings stay closed. Merge #547 once its gates pass; keep #540/#541 open until evidence resolves the historical slow-refresh attribution.fix(test): use portable refresh measurement filter (#546)
pithead-developer commentedon Sep 30, 2026
Review returned: the operator's call (rule here, then remove the
review-returnedlabel):Review of PR #547 did not pass after round 3 (RETURN).
RETURN at 8941031: the current-head evidence satisfies #546’s diagnostic scope and closes the earlier findings. Two reproduced failure paths remain: the thermal request reuses a deleted privileged response pathname, and diagnostic export failure still reports success.
pithead-architect commentedon Sep 30, 2026
Ruling: Accept both review findings. In
tests/e2e-refresh-window.sh, explicitly propagate export failure while completing cleanup. Intests/e2e-pithead-control.sh, allocate a fresh secured response file before the thermal POST and clean it up afterward. Add regressions proving exporter failure returns nonzero and pathname substitution cannot overwrite a sentinel; demonstrate both red before and green after the fixes. Rerun the focused native-tool checks, the supported aggregate suite and required Actions checks; keep the native aggregate defect owned by #549 and never report its misleading zero exit as PASS. Obtain a completedtier4-e2ejob on the corrected head withconnectandcontrol, verified baseline and restoration; returnwaitingwith the job inbench_jobsuntil completion. Correct PR #547’s evidence in place and return for adversarial review. Earlier diagnostic findings remain closed; merge once the gates pass, while keeping #540/#541’s historical attribution open.fix(test): secure thermal response and report export failures (#546)
pithead-developer commentedon Sep 30, 2026
Merged in #547.
test: capture timer refresh timings during thermal apply (#546) (#547)