perf(api): avoid per-core processes in refresh (#540)

docs/operations.md

@@ -564,8 +564,12 @@ last-change provenance (`config_meta`; #254), and the last control-path outcome
 `{change_id, status, reason}` or `null`; #346) — as JSON (`/health`, or nested under
 `rigforge` in `/1/summary` and `/2/summary`). It follows XMRig's own architecture: one tiny
 persistent server (python3 stdlib, ~10 MB idle) ships pre-computed bytes, so a request costs
-microseconds and cannot touch mining performance; a systemd timer recomputes the state every 15
-seconds at idle priority, off the request path. The timer uses an independent wall-clock cadence,
+microseconds and cannot touch mining performance; a systemd timer schedules a refresh every 15
+seconds off the request path. Refreshes use normal CPU priority (`Nice=0`) to share CPU time with
+the miner when publishing control changes; disk I/O stays at idle priority and
+the cached HTTP server retains `Nice=19`. The queued thermal and separate pools replay tests check direct-feed
+freshness and exact-ID publication, capturing timer-driven helper timings under the runner-owned
+mining fixture. They preserve the service policy and normal feed publication. The timer uses an independent wall-clock cadence,
 so a failed refresh or a unit reinstall cannot strand the feed without a future run. Summary and
 health responses carry `generated_at` in UTC; `doctor` reports its age and the timer's next/last
 trigger, and warns once the payload is over a minute old. The same

docs/pithead-integration.md

@@ -111,7 +111,9 @@ pool, `:8081` exists exactly while enabled, and no response byte ever contains `
 pool `pass`. `:8080` stays the canonical Pithead summary probe; `:8081` is additive. Port/bind are
 `api_port`/`api_bind`. Architecture mirrors XMRig's own API: a tiny persistent server ships
 pre-computed state (a request costs microseconds — polling cannot shave hashrate), refreshed every
-15s by an idle-priority wall-clock timer. Consumers should still age `generated_at`: it distinguishes
+15s by a wall-clock timer. Refresh work uses normal CPU priority with idle disk I/O to give
+control-change publication a normal CPU share during mining; the cached HTTP server remains at idle CPU
+priority. Consumers should still age `generated_at`: it distinguishes
 a current report from a cached response when a rig or its refresh job is unhealthy.
 
 ### Stratum over TLS (optional)

systemd/rigforge-api-refresh.service.template

@@ -3,12 +3,12 @@ Description=RigForge sister API refresh (probe pass -> /run/rigforge-api, off th
 
 [Service]
 Type=oneshot
-# Root for the probes (dmidecode / rdmsr / RAPL); idle priority — a background refresh has no
-# latency requirement, so it must never compete with the miner.
+# Root for the probes (dmidecode / rdmsr / RAPL). Give control-change publication a normal
+# CPU share during mining; disk I/O remains idle priority.
 Environment=RIGFORGE_OPERATOR=$RIGFORGE_OPERATOR
 ExecStart=$SCRIPT_DIR/rigforge.sh api-refresh
 TimeoutStartSec=300
-Nice=19
+Nice=0
 IOSchedulingClass=idle
 RuntimeDirectory=rigforge-api
 RuntimeDirectoryPreserve=yes

tests/README.md

@@ -28,8 +28,14 @@ against GitHub Actions' ToS. They're a manual pre-tag gate the releaser runs.
 The queued `control` phase also instruments real timer-driven API refreshes around a thermal
 control apply (#546). A temporary runtime drop-in replaces only the refresh service's command
 with `e2e-api-refresh-profile.sh`; its priority, timeout, timer and live feed remain in use.
-The profiler logs fixed helper names, begin/end timestamps and nanosecond durations, never
-arguments or payloads. Five-second observations record timer/service state and the direct feed
+The profiler logs fixed helper names, begin/end timestamps and nanosecond durations, plus full-pass
+CPU ticks (the shell and waited-for children), clock ticks per second, nice level and the shell's
+cumulative scheduler wait in nanoseconds. CPU-time deltas distinguish computation from elapsed
+waiting; scheduler wait excludes children and can be zero when kernel scheduler statistics are
+disabled, so zero does not prove absence of contention. Missing/malformed accounting or a journal
+without a successful refresh and CPU begin/end from the same process fails diagnostics.
+No arguments, process names or payloads are exported.
+Five-second observations record timer/service state and the direct feed
 for five minutes around the apply; the changed ceiling must still appear within 90 seconds.
 Control-phase requests pass the temporary bearer token through curl's stdin configuration,
 keeping it out of process arguments. Each POST uses its own securely allocated response file,
@@ -42,11 +48,34 @@ the owned drop-in, reloads systemd and restores the timer’s prior active state
 The job fails if journal collection or measurement export fails, no profiled refresh completes
 successfully, or cleanup fails. Begin events from unfinished helpers are retained to locate stalled
 operations, including a pass stopped by cleanup.
-For affected-workload acceptance, use a reserved `rigforge` `tier4-e2e` job with
-`phases=["connect","control"]`: connect proves mining before control on the same reserved worker.
-A `phases=["control"]` run validates diagnostic collection only;
-do not invoke the profiler against a rig yourself. Routing still chooses an eligible worker:
-this captures the same thermal-control/service path, but does not guarantee the original fixture.
+The ordinary `phases=["connect","control"]` job proves mining and profiles a bounded thermal
+apply on an eligible worker; it does not recreate the historical sequence. For #540's separate
+thermal and pools replay cases, submit one declaring phase per `tier4-e2e` job with
+`fixture=pithead-control`: `control-replay-thermal` or `control-replay-pools`. The deployed runner
+privately supplies the operator pools probe, context and named reservation. The gate verifies that
+reservation, bypasses only its own competing flock, and leaves the runner's holder marker and release
+alone. The hardware-free `test-control-replay.sh` uses Bash builtins for its apply fixture and checks that refusal before config changes and the
+unchanged ordinary locking; on failure it prints each synthetic case log's final 20 lines before
+removing its sandbox, preserving the failed exit status.
+Both cases establish real jobs and an accepted share, retain thread/tuning and payout identity, set
+recorded scalars (thermal 100, DONATION 0, watchdog interval 5), and replay thermal 101 → 100,
+DONATION 1 → 0, watchdog interval 6 → 5, then the private pools probe. The thermal case verifies a 100 baseline, then applies direct 102 and requires a newer feed
+carrying that cutoff and this exact terminal change ID within 90 seconds. The separate pools
+case requires this exact change ID to reach `applied` in the enriched feed within the same bound,
+not a later change's outcome. Both continue five-minute observations for diagnostics and preserve the
+original pass/fail deadline even if a read crosses it. Full applies can stop/restart the miner;
+observations record its active state rather than aborting during that expected transition.
+Status and summary GETs consume JSON only after a successful transport; failed-read bytes are
+discarded and a fixed transport-failure marker is emitted. Valid JSON followed by a nonzero
+transport exit cannot satisfy settlement or publication. Each apply settlement emits only
+elapsed seconds and its validated status. Mining-precondition
+failures emit fixed-schema systemd state plus counts of jobs, shares and classified XMRig errors;
+unknown service values fail the diagnostic, and no journal text, endpoint or config value is printed.
+Requests use secured scratch payloads, never credential-bearing arguments or logs. The current probe
+and borrowed worker form an equivalent fixture: historical starting config/checkout, original pool
+payload, threads/affinity/tuning and precise inter-apply timing remain unknown in the runner context.
+No guessed delays or hardware differences are substituted. Do not invoke any replay or profiler
+against a worker yourself; the runner owns reservation, restoration and baseline verification.
 Short passes or an injected delay do not attribute the historical stale-feed failure; #540's
 fix acceptance requires evidence of the same failure mechanism. Profiling adds clock-read overhead
 and the source remains the submitted RigForge head; no alternative production implementation is

tests/e2e-api-refresh-profile.sh

@@ -4,6 +4,23 @@
 set -euo pipefail
 # shellcheck disable=SC1090
 source "$1"
+PROFILE_CLOCK_TICKS=$(getconf CLK_TCK)
+[[ "$PROFILE_CLOCK_TICKS" =~ ^[1-9][0-9]*$ ]]
+refresh_profile_cpu() {
+    local stat runtime wait slices field profile_pid="${BASHPID:-$$}"
+    local -a fields
+    IFS= read -r stat <"${REFRESH_PROFILE_STAT_FILE:-/proc/$profile_pid/stat}" || return 1
+    # comm may contain spaces or parentheses; every field after its final ')' is numeric/state.
+    read -r -a fields <<<"${stat##*) }"
+    read -r runtime wait slices <"${REFRESH_PROFILE_SCHEDSTAT_FILE:-/proc/$profile_pid/schedstat}" || return 1
+    for field in "${fields[11]:-}" "${fields[12]:-}" "${fields[13]:-}" "${fields[14]:-}" "$runtime" "$wait" "$slices"; do
+        [[ "$field" =~ ^[0-9]+$ ]] || return 1
+    done
+    [[ "${fields[16]:-}" =~ ^-?[0-9]+$ ]] && [ "${fields[16]}" -ge -20 ] && [ "${fields[16]}" -le 19 ] || return 1
+    # CPU ticks include this shell and waited-for children; scheduler delay is this shell only.
+    printf 'refresh-profile-cpu: pid=%s event=%s cpu_ticks=%s clock_ticks_per_s=%s nice=%s scheduler_wait_ns=%s\n' \
+        "$profile_pid" "$1" "$((10#${fields[11]} + 10#${fields[12]} + 10#${fields[13]} + 10#${fields[14]}))" "$PROFILE_CLOCK_TICKS" "${fields[16]}" "$wait" >&2
+}
 refresh_profile_event() {
     printf 'refresh-profile: pid=%s helper=%s event=%s at_ns=%s duration_ns=%s rc=%s\n' \
         "$$" "$1" "$2" "$3" "$4" "$5" >&2
@@ -16,8 +33,10 @@ for probe in api_refresh parse_config _read_api_summary _api_rigforge_block _api
         local profile_start profile_end profile_rc
         profile_start=\$(date +%s%N)
         refresh_profile_event $probe begin \"\$profile_start\" 0 0
+        [ $probe != api_refresh ] || refresh_profile_cpu begin
         ${probe}_profile_original \"\$@\"
         profile_rc=\$?
+        [ $probe != api_refresh ] || refresh_profile_cpu end
         profile_end=\$(date +%s%N)
         refresh_profile_event $probe end \"\$profile_end\" \"\$((profile_end - profile_start))\" \"\$profile_rc\"
         return \"\$profile_rc\"