Skip to content

bench-ci nightly: tier4-e2e failure on develop@a509635 #545

Closed as not planned
Closed as not planned
bench-ci nightly: tier4-e2e failure on develop@a509635#545
Labels
bench-ciFiled by a bench-ci job: a nightly or gate failure

Description

@VijitSingh97
VijitSingh97
opened on Sep 30, 2026

Automated by bench-ci job 292 (nightly, submitted by nightly).
That id is local to the bench that ran it: resolve it with bench-ci list --fleet --commit a50963583fa92c0b58b675560f264c46b6107484, then bench-ci status <id> --bench <the bench that row names>.

  • commit: a50963583fa92c0b58b675560f264c46b6107484 ()
  • tier: tier4-e2e, options {"phases": ["all"]}
  • created 2026-09-30T06:00:05.543Z, started 2026-09-30T06:00:17.053Z, finished 2026-09-30T06:03:08.285Z
  • conclusion: failure, exit code 1
  • steps: checkout=success, reserve=success, preflight=success, run=failure, collect=success, restore=success, cleanup=success
  • baseline verified afterwards: True

Last lines of the job log (redacted):

2026-09-30T06:00:50.637Z [run] --- run ---
2026-09-30T06:00:50.882Z [run] rig lock on <bench> handed to the harness
2026-09-30T06:00:51.223Z [run] == fetch and check out a50963583fa92c0b58b675560f264c46b6107484 in the rig's install tree (/opt/rigforge) ==
2026-09-30T06:00:53.052Z [run] == e2e-pithead all ==
2026-09-30T06:00:53.356Z [run] 
2026-09-30T06:00:53.356Z [run] \x1b[1m== e2e-pithead: connect — worker mines against the live stack (<bench-stratum>) ==\x1b[0m
2026-09-30T06:02:27.591Z [run]   \x1b[1;31m✗\x1b[0m no stratum job from <bench-stratum> within 60s — is the stack up and reachable?
2026-09-30T06:02:41.294Z [run] phase all: FAILED (rc=1)
2026-09-30T06:02:41.294Z [run] == restoring the rig's checkout to 860c8d1b1d6469a8a92f2875db5a1a232c0dd809 and re-applying ==
2026-09-30T06:02:56.508Z [run] exit 1

Filed by the bench runner; a repeat failure of this tier on the same commit is added as a comment.

Activity

  1. VijitSingh97
    added
    bench-ciFiled by a bench-ci job: a nightly or gate failure
    on Sep 30, 2026

  2. VijitSingh97 commented on Sep 30, 2026

    @VijitSingh97
    VijitSingh97
    on Sep 30, 2026
    ContributorAuthor

    Automated by bench-ci job 163 (nightly, submitted by nightly).
    That id is local to the bench that ran it: resolve it with bench-ci list --fleet --commit a50963583fa92c0b58b675560f264c46b6107484, then bench-ci status <id> --bench <the bench that row names>.

    • commit: a50963583fa92c0b58b675560f264c46b6107484 (chore(ci): update stale tool pins before 2.0.0 (chore(ci): update stale tool pins before 2.0.0 #543))
    • created 2026-09-30T06:00:14.081Z, started 2026-09-30T06:00:15.601Z, finished None
    • conclusion: None
    • failing step: apply — rigforge.sh apply failed or the miner did not reach a pool afterwards (apply-new.log in the artifacts)
    • rollback: rolled back to 860c8d1 and verified healthy
    • baseline verified afterwards: None

    Last lines of the job log (redacted):

    2026-09-30T06:00:15.765Z [run] --- run ---
2026-09-30T06:00:16.110Z [run] == /opt/rigforge: 860c8d1b1d6469a8a92f2875db5a1a232c0dd809 -> a50963583fa92c0b58b675560f264c46b6107484 ==
2026-09-30T06:00:16.110Z [run] == previous build: measuring the hashrate baseline ==
2026-09-30T06:00:39.667Z [run] baseline: miner up, [2026-09-30 06:00:38.086]  net      new job from pithead-prod:3333 diff 250012 algo rx/0 height 3773
2026-09-30T06:00:39.667Z [run] baseline: settling 180s, then sampling 60s
2026-09-30T06:04:39.956Z [run] baseline: 30927.7 H/s on 860c8d1b1d6469a8a92f2875db5a1a232c0dd809
2026-09-30T06:04:57.688Z [run] [INFO] Configuring log rotation policy...
2026-09-30T06:04:57.688Z [run] [INFO] Applied config and restarted xmrig.
2026-09-30T06:04:57.688Z [run] [INFO] API firewall active — port(s) 8080, 8081, 8082 reachable only from <redacted>/24 and loopback.
2026-09-30T06:04:57.688Z [run] [INFO] Periodic autotune: disabled.
2026-09-30T06:04:57.688Z [run] [INFO] Miner reports a live pool connection to pithead-prod:3333.
2026-09-30T06:05:16.064Z [run] new: xmrig died before reaching a pool (it never came up)
2026-09-30T06:05:16.064Z [run] FAIL [apply]: rigforge.sh apply failed or the miner did not reach a pool afterwards (apply-new.log in the artifacts)
2026-09-30T06:05:16.064Z [run] == rolling back to 860c8d1b1d6469a8a92f2875db5a1a232c0dd809 ==
2026-09-30T06:05:32.883Z [run] [INFO] Configuring log rotation policy...
2026-09-30T06:05:32.883Z [run] [INFO] Applied config and restarted xmrig.
2026-09-30T06:05:32.883Z [run] [INFO] API firewall active — port(s) 8080, 8081, 8082 reachable only from <redacted>/24 and loopback.
2026-09-30T06:05:32.883Z [run] [INFO] Periodic autotune: disabled.
2026-09-30T06:05:32.884Z [run] [INFO] Miner reports a live pool connection to pithead-prod:3333.
2026-09-30T06:05:46.324Z [run] rollback: miner up, [2026-09-30 06:05:43.929]  net      use pool pithead-prod:3333  <redacted>
2026-09-30T06:05:46.325Z [run] exit 1

    Filed by the bench runner; a repeat failure on the same commit is added as a comment.

  3. VijitSingh97 commented on Sep 30, 2026

    @VijitSingh97
    VijitSingh97
    on Sep 30, 2026
    ContributorAuthor

    Automated by bench-ci job 119 (nightly, submitted by nightly).
    That id is local to the bench that ran it: resolve it with bench-ci list --fleet --commit a50963583fa92c0b58b675560f264c46b6107484, then bench-ci status <id> --bench <the bench that row names>.

    • commit: a50963583fa92c0b58b675560f264c46b6107484 (chore(ci): update stale tool pins before 2.0.0 (chore(ci): update stale tool pins before 2.0.0 #543))
    • created 2026-09-30T06:00:49.248Z, started 2026-09-30T06:00:51.070Z, finished None
    • conclusion: None
    • failing step: apply — rigforge.sh apply failed or the miner did not reach a pool afterwards (apply-new.log in the artifacts)
    • rollback: rolled back to 860c8d1 and verified healthy
    • baseline verified afterwards: None

    Last lines of the job log (redacted):

    2026-09-30T06:00:51.232Z [run] --- run ---
2026-09-30T06:00:51.581Z [run] == /opt/rigforge: 860c8d1b1d6469a8a92f2875db5a1a232c0dd809 -> a50963583fa92c0b58b675560f264c46b6107484 ==
2026-09-30T06:00:51.581Z [run] == previous build: measuring the hashrate baseline ==
2026-09-30T06:01:05.023Z [run] baseline: miner up, 
2026-09-30T06:01:05.023Z [run] baseline: settling 180s, then sampling 60s
2026-09-30T06:05:05.301Z [run] baseline: 31288.5 H/s on 860c8d1b1d6469a8a92f2875db5a1a232c0dd809
2026-09-30T06:05:23.953Z [run] [INFO] Configuring log rotation policy...
2026-09-30T06:05:23.953Z [run] [INFO] Applied config and restarted xmrig.
2026-09-30T06:05:23.953Z [run] [INFO] API firewall active — port(s) 8080, 8081, 8082 reachable only from <redacted>/24 and loopback.
2026-09-30T06:05:23.953Z [run] [INFO] Periodic autotune: disabled.
2026-09-30T06:05:23.953Z [run] [INFO] Miner reports a live pool connection to pithead-prod:3333.
2026-09-30T06:05:37.076Z [run] new: xmrig died before reaching a pool (it never came up)
2026-09-30T06:05:37.076Z [run] FAIL [apply]: rigforge.sh apply failed or the miner did not reach a pool afterwards (apply-new.log in the artifacts)
2026-09-30T06:05:37.076Z [run] == rolling back to 860c8d1b1d6469a8a92f2875db5a1a232c0dd809 ==
2026-09-30T06:05:52.056Z [run] [INFO] Configuring log rotation policy...
2026-09-30T06:05:52.056Z [run] [INFO] Applied config and restarted xmrig.
2026-09-30T06:05:52.057Z [run] [INFO] API firewall active — port(s) 8080, 8081, 8082 reachable only from <redacted>/24 and loopback.
2026-09-30T06:05:52.057Z [run] [INFO] Periodic autotune: disabled.
2026-09-30T06:05:52.057Z [run] [INFO] Miner reports a live pool connection to pithead-prod:3333.
2026-09-30T06:06:10.712Z [run] rollback: miner up, 
2026-09-30T06:06:10.713Z [run] exit 1

    Filed by the bench runner; a repeat failure on the same commit is added as a comment.

  4. pithead-developer commented on Oct 1, 2026

    @pithead-developer
    pithead-developerbot
    on Oct 1, 2026
    Contributor

    Additional evidence from RigForge #540, draft PR #541: jobs 1978, 1979 and 1980 at c9c4ce0 all failed to observe a Stratum job. The replay jobs also observed no accepted share; the API impact job could not obtain a live hashrate baseline. Full statuses, logs, inventories and every artifact were read. Both replay jobs sampled an active miner service throughout; their evidence does not identify a stack failure versus a miner connection failure. Restoration and baseline verification succeeded.

    I also read the full retained status/log and sole artifact of 292@a50963583fa92c0b58b675560f264c46b6107484, confirming that this assertion failed on develop before the scheduling candidate. A common underlying cause remains unproven. A bounded diagnostic extension in #541 will collect allowlisted miner state/error counters and scalar-apply settlement status through the approved queue; no manual hardware action is being taken.

  5. VijitSingh97 commented on Oct 3, 2026

    @VijitSingh97
    VijitSingh97
    on Oct 3, 2026
    ContributorAuthor

    Superseded. The 'no stratum job within 60s' failure was traced to the apply failures fixed under #540; later nightlies get past connect (see #563). The remaining stratum-auth work is rigforge#562 and bench-ci#1127.

  6. VijitSingh97
    closed this as not plannedon Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bench-ciFiled by a bench-ci job: a nightly or gate failure

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions