Repository navigation
bench-ci nightly: tier4-e2e failure on develop@a625d1d #563
Description
What's the problem
Automated by bench-ci job 349 (nightly, submitted by nightly).
That id is local to the bench that ran it: resolve it with bench-ci list --fleet --commit a625d1d497978d320a6e1b01125717fdbec667ca, then bench-ci status <id> --bench <the bench that row names>.
- commit:
a625d1d497978d320a6e1b01125717fdbec667ca() - tier:
tier4-e2e, options{"phases": ["all"]} - created 2026-10-02T06:00:05.444Z, started 2026-10-02T06:00:17.113Z, finished 2026-10-02T06:05:10.920Z
- conclusion:
failure, exit code 1 - steps: checkout=success, reserve=success, preflight=success, run=failure, collect=success, restore=success, cleanup=success
- baseline verified afterwards: True
Last lines of the job log (redacted):
2026-10-02T06:00:52.294Z [run] --- run ---
2026-10-02T06:00:53.247Z [run] rig lock on <rig> handed to the harness
2026-10-02T06:00:53.568Z [run] == fetch and check out a625d1d497978d320a6e1b01125717fdbec667ca in the rig's install tree (/opt/rigforge) ==
2026-10-02T06:00:55.997Z [run] == e2e-pithead all ==
2026-10-02T06:00:56.300Z [run]
2026-10-02T06:00:56.300Z [run] == e2e-pithead: connect — worker mines against the live stack (<bench-stratum>) ==
2026-10-02T06:01:19.372Z [run] ✓ connected — stratum job from <bench-stratum>
2026-10-02T06:01:39.423Z [run] ✓ share accepted by the stack (1 so far)
2026-10-02T06:01:39.436Z [run] ✓ stratum user label set ('<tier-input>') — what the dashboard discovers by
2026-10-02T06:01:39.447Z [run]
2026-10-02T06:01:39.447Z [run] == e2e-pithead: worker-api — the :8080 contract (open read-only by default; Bearer when ACCESS_TOKEN set) ==
2026-10-02T06:02:02.518Z [run] ✓ open posture: /2/summary readable with no token
2026-10-02T06:02:02.535Z [run] ✓ API bound to <ipv4> (reachable from the stack host)
2026-10-02T06:02:02.547Z [run] ✓ restricted: PUT /1/config refused (403)
2026-10-02T06:02:16.231Z [run] ✓ unauthenticated read returned HTTP 401
2026-10-02T06:02:16.245Z [run] ✓ matching credentials returned HTTP 200
2026-10-02T06:02:33.567Z [run]
2026-10-02T06:02:33.567Z [run] == e2e-pithead: api-impact — the sister API must not shave hashrate (#99 perf guard) ==
2026-10-02T06:02:48.446Z [run] ✓ sister API serving on :8081
2026-10-02T06:04:05.881Z [run] ✓ hashrate under API hammering within 3% of baseline (36823.0 -> 36804.7 H/s)
2026-10-02T06:04:06.202Z [run] ✓ sister API answers /health in 0s under full mining load (budget 15s)
2026-10-02T06:04:16.844Z [run]
2026-10-02T06:04:16.844Z [run] == e2e-pithead: network — nothing listens or leaks beyond what the config defines ==
2026-10-02T06:04:17.148Z [run] ✓ outbound: xmrig's only TCP peers are the pool (:3333)
2026-10-02T06:04:31.134Z [run] ✓ :8081 listening from control alone, api never set (#507)
2026-10-02T06:04:41.779Z [run] ✓ :8081 listening while api enabled
2026-10-02T06:04:41.791Z [run] ✓ miner listens on :8080 only
2026-10-02T06:04:42.105Z [run] ✗ sister summary lacks valid metadata or its XMRig key superset differs
2026-10-02T06:04:52.134Z [run] phase all: FAILED (rc=1)
2026-10-02T06:04:52.134Z [run] == restoring the rig's checkout to a625d1d497978d320a6e1b01125717fdbec667ca and re-applying ==
2026-10-02T06:05:04.884Z [run] exit 1
Filed by the bench runner; a repeat failure of this tier on the same commit is added as a comment.
what it impacts
Impact is not separately established in the existing report; confirm the affected users or behavior.
How I found it
Discovery details are not separately established in the existing report; preserve and verify the evidence already recorded.
how important is it
Importance has not been assessed explicitly; triage must state the priority and its reason.
acceptance criteria
Acceptance is not isolated in the existing report; the agent must identify a testable outcome before implementation.
possible solutions
No distinct solution options are recorded; investigate the existing report before choosing an approach.
Exact original fixture values and log text are preserved in private runner notes; this formatting update ran no new checks.
Activity
pithead-developer commentedon Oct 3, 2026
Blocked on bench-ci#1144: approved CI list and narrow reproduction submission now fail with connection refused. No job id was returned, so no hardware result or passing reproduction is claimed. Bench maintainers must restore endpoint availability; resume the same
networksubmission at7088ca6a402c0910519d81013c799b848709d737with idempotency key9faf250079254db39bd32d5175ff9258to safely recover any accepted job.The bounded diagnostic change, required focused regression (22 assertions), documentation and author-side verifier/security reviews are complete in draft PR #568. Full historical job logs and the sole artifact were read, but they lack the observations needed to establish the cause. Root-cause diagnosis and correction remain unfinished. Once a recovered/new diagnostic job completes, read its full logs/artifacts, renew generated evidence and continue on this same PR. No manual bench or rig action was performed.
pithead-developer commentedon Oct 3, 2026
Blocked on p2pool-starter-stack/bench-ci#1144. Draft PR #568 adds bounded summary diagnostics; 22 regression assertions pass and verifier/security reviews are clean. Bench-ci#1144 blocks reproduction because the approved endpoint refuses connections; maintainers must restore access, then recover the submission using its original idempotency key. Root-cause diagnosis and correction remain unfinished, with the blocker documented on the issue and PR.
The runner resumes this issue when that one closes.
pithead-developer commentedon Oct 3, 2026
The operator closed bench-ci#1144 after restoring endpoint availability. The original idempotency key resumed submission as job
2052@7088ca6a402c0910519d81013c799b848709d737, restricted tonetwork. All eight current-head GitHub checks passed. PR #568 remains draft and awaits the diagnostic job; root-cause correction is still outstanding. Generated evidence is in its body; the queued identifier mismatch is reported on existing bench-ci#922 and will be checked again after completion.test: reproduce nightly prefix in a narrow network phase (#563)
pithead-developer commentedon Oct 3, 2026
The completed isolated network job
2052@7088ca6a402c0910519d81013c799b848709d737passed; its full status, log and sole artifact were audited. It does not reproduce the preceding nightly phase state. PR #568 now adds a narrownetwork-sequencereproduction, sharing one process/snapshot across the four-phase nightly prefix and omitting unrelated later phases. Focused gate regressions and author-side verifier/security reviews passed. Job2054@48b227098a8eae72503de741de18d12d740fe6b3is queued for that sequence; root-cause correction remains unfinished. Refreshed unedited evidence for every job is in the PR body.pithead-developer commentedon Oct 3, 2026
Audited completed job
2054@48b227098a8eae72503de741de18d12d740fe6b3: full status, complete log and sole retained phase artifact. The nightly prefix passed, including the unchanged summary metadata/key assertion, with valid diagnostic flags and verified restoration. All eight required GitHub checks passed at this head; focused regressions and author-side reviews are clean. Every generated evidence block in draft PR #568 has been refreshed unedited.No product code or strict assertion was changed, and the original cause remains unproved. This issue retains the intermittent failure and is also labeled
flaky; no further rerun loop is being started. PR #568 is complete as bounded diagnostic instrumentation and a narrow reproduction phase, but it cannot honestly be claimed to resolve the original failure. It remains draft pending operator disposition of that distinction. No merge-readiness claim is made for closing #563.pithead-developer commentedon Oct 3, 2026
Filed privately (rigforge-563.md); it needs an answer there, then remove the
questionlabel.pithead-developer commentedon Oct 3, 2026
The operator split the tested diagnostic work into #569 on
maintenance & gates, reusing PR #568. Its closing reference now targets only #569; #563 remains open onv2 - appliancefor the unresolved intermittent failure.#563 is blocked on #569 landing: the diagnostic code must be available to the scheduled nightly before a recurrence can supply the missing observations. The implementation, mandatory regressions, documentation, author-side reviews, all eight current-head GitHub checks and audited tier4-e2e job
2054@48b227098a8eae72503de741de18d12d740fe6b3are finished for the prerequisite. PR #568 has been submitted for the separate adversarial-review gate; author-side reviews do not replace it. The independent reviewer must record a passing status at the tested head and the runner must land the prerequisite. That clears this dependency; afterward the next scheduled rigforge nightly is the evidence source, and only a matching failure resumes causal diagnosis. No speculative rerun is being started and no passing reproduction closes #563 as fixed.pithead-developer commentedon Oct 3, 2026
Blocked on #569. Created diagnostics prerequisite #569 on maintenance & gates and verified PR #568 closes only that issue. The unchanged, tested head is submitted for separate adversarial review; #563 is blocked until the reviewer approves and the runner lands #569. Afterward, the next scheduled nightly supplies recurrence evidence; #563 remains open on v2 - appliance.
The runner resumes this issue when that one closes.
VijitSingh97 commentedon Oct 3, 2026
Owner: moved out of 2.0.0 with #569. I pulled #563 into v2 at 05:50Z only because the 10-02 nightly might have been a regression in the RigForge code the 2.0.0 image bakes (
9b05e76). Both targeted repros since then, jobs 2052 and 2054, passed the network sequence. A failure that does not reproduce is a soak watch under the 09-27 scope rule, as #2815 and #2946 were. This moves back into v2 if it recurs in the gated run on the frozen SHA or in the soak. The diagnostics work (#569 / PR #568) can still land on develop; v1.18.0 is tagged atf5ff047regardless (#564).test: retain network summary diagnostics and prefix reproduction (#569)…
pithead-developer commentedon Oct 12, 2026
Diagnostics prerequisite #569 is complete through merged PR #568, merge commit
0b04f2a50dfff2286288371c7b87f354672ce768. The implementation dependency is cleared; #563 remains unresolved. The later owner ruling keeps this issue onmaintenance & gatesas a soak watch unless the failure recurs in the gated run or soak.No open PR currently owns #563. A bounded fleet listing of the latest 100 rigforge tier4-e2e jobs contains no post-merge completed failure/error with the exact failed row retained from
349@a625d1d497978d320a6e1b01125717fdbec667ca:sister summary lacks valid metadata or its XMRig key superset differsFollowing the standing operator instruction, await new diagnostic evidence from the scheduled rigforge
tier4-e2enightly. The registered recurrence trigger is a new completed failure/error job in that source with this exact row marked fail. Passing jobs and unrelated failed rows do not trigger causal diagnosis. On a matching recurrence, audit the full status, complete log and all artifacts, including the merged bounded diagnostics, before inferring a cause. No speculative rerun or issue closure is being performed.