Skip to content

fix(sp-update): skip biometrics-archiver setup on new (NATS) firmware + diagnostics#594

Open
ng wants to merge 5 commits into
devfrom
worktree-pod5-debug
Open

fix(sp-update): skip biometrics-archiver setup on new (NATS) firmware + diagnostics#594
ng wants to merge 5 commits into
devfrom
worktree-pod5-debug

Conversation

@ng

@ng ng commented May 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Pod 5 firmware ca35aafa (April 2026+) writes biometric CBOR frames to NATS JetStream stream raw (subject raw.>) at /persistent/jetstream/ instead of .RAW files. Previously install_biometrics_archiver continued past the missing frank.sh shim and tripped sp-update's set -euo pipefail on a subsequent systemctl enable --now <timer>, rolling back the OTA. Symptom: Warning: /opt/eight/bin/frank.sh not found — skipping patch (not a Pod?) immediately followed by Update failed, rolling back....

This PR:

  1. Gates the helper on new-firmware NATS signals (nats-server.service active AND /persistent/jetstream directory present). Only those pods skip; mid-era pods (frank.sh absent, no NATS — exists for ~year now) keep the existing warn-and-continue path so their archive/prune rotation isn't lost.
  2. Expands sp-status from a one-liner into a full diagnostic: firmware variant, biometrics pipeline (old/mid/new), NATS probe (TCP 4222 + nats stream info raw), JetStream dir size, .RAW counts at each location, sleepypod modules, biometrics-archiver units, firmware-side services. Output is paste-friendly for Discord support threads.
  3. Adds scripts/probe-nats-capture.py — a one-shot Python recorder (uv shebang, pulls nats-py + cbor2 on first run) that subscribes to a NATS subject filter, dumps every message as NDJSON with CBOR-decoded shape, and prints a per-subject histogram on exit. Used to discover what subjects/schemas the new firmware actually publishes before designing the consumer.
  4. Documentation: ADR 0018 gets a firmware-variant applicability matrix; the biometrics wiki and scripts/README.md describe the new data source path and the discovery probe.

This is the OTA-unblock stopgap. Actual biometrics ingestion against NATS JetStream is tracked separately in sleepypod-core-54 (epic) with subtasks 55 (discovery via the new probe), 56 (BiometricsSource abstraction + nats-py source), 57 (status surface). Per the saved feedback memory, that work must be additiveRawFileFollower stays the default and old/mid pods are unaffected.

Firmware-variant matrix

Era frank.sh nats-server /persistent/jetstream Biometric writes Helper behavior
Old (Pod 3 / Pod 4 / early Pod 5) present absent absent firmware via shim → /persistent/*.RAW unchanged: patch shim, mount tmpfs, install timers
Mid (~2025 → April 2026) absent absent absent firmware direct → /persistent/*.RAW unchanged: warn-skip patch, mount + archive loop still runs
New (April 2026+) absent active present firmware → NATS subjects skip entirely (this PR)

Verification

Local harness with stubbed systemctl/install/mountpoint exercises all four control-flow branches (new fw, mid-era, old fw, anomaly = NATS up but no jetstream dir → conservative fall-through to legacy install). All pass. Shell syntax + shellcheck clean.

Test plan

  • Pod 5 with new firmware (ca35aafa+, nats-server active, /persistent/jetstream/ populated): sp-update completes without rollback; log shows new firmware detected (nats-server active; /persistent/jetstream present) — skipping tmpfs setup. sp-status reports pipeline: NATS JetStream (new firmware, April 2026+).
  • Pod with mid-era firmware (frank.sh absent, no NATS): sp-update continues to install mount + timers + archive loop as it does today — no regression in .RAW rotation at /persistent. sp-status reports pipeline: direct *.RAW (mid-era firmware; ...).
  • Pod 4 with old firmware (frank.sh present): sp-update patches frank.sh, mounts tmpfs, installs timers — completely unchanged path. sp-status reports pipeline: tmpfs *.RAW via frank.sh shim (old firmware).
  • sleepypod.service starts and stays active after the OTA on each pod variant.
  • On new-firmware pod: scripts/probe-nats-capture.py --duration 30 produces an NDJSON dump with non-zero message count and per-subject histogram. CBOR decode succeeds for at least one subject.

Known follow-up (out of scope)

After this PR, new-firmware pods OTA cleanly but do not ingest biometric data — the NATS consumer is sleepypod-core-54 epic. Discovery captures (sleepypod-core-55) gate the implementation work; probe-nats-capture.py exists for that purpose.

🛏️ Beam this PR onto a sleepypod

Pick the snippet that matches your pod. Already-installed pods can't use the curl bootstraps below — sleepypod's egress firewall DROPs github.com, and the script that knows how to unblock WAN is the very file curl is trying to fetch. Use sp-update instead; it's on disk and opens WAN as its first step.

🔄 Already on a sleepypod (most common — review a PR on a running pod):

sudo sp-update worktree-pod5-debug

🚀 Fresh pod / first install — bootstraps from GitHub, rebuilds every push:

curl -fsSL https://raw.githubusercontent.com/sleepypod/core/worktree-pod5-debug/scripts/install \
  | sudo bash -s -- --branch worktree-pod5-debug

🎯 Pin to this exact build (run 26003896986 · 6e3536c) — fresh-pod path, locked to one CI artifact for reproducible review:

curl -fsSL https://raw.githubusercontent.com/sleepypod/core/6e3536c652bc144c2341d8b1813c1f0f0c1179de/scripts/install \
  | sudo bash -s -- \
      --branch worktree-pod5-debug \
      --artifact-url 'https://nightly.link/sleepypod/core/actions/runs/26003896986/sleepypod-core.zip'

🧊 Artifact: sleepypod-core · self-destructs in 30 days 💥

Pod 5 firmware ca35aafa (April 2026+) refactored away from
/opt/eight/bin/frank.sh — biometric frames now go to NATS JetStream
at /persistent/jetstream/ instead of /persistent/*.RAW. The helper
previously warned and continued, then a subsequent systemctl enable
failed under sp-update's set -euo pipefail and rolled back the OTA.

Use frank.sh as the canary for old-firmware applicability; return 0
cleanly when absent so OTAs succeed on the new firmware. The defensive
warn-and-skip in the patch block stays in as a belt-and-suspenders
guard for future code paths.
@coderabbitai

coderabbitai Bot commented May 17, 2026

Copy link
Copy Markdown
📝 Walkthrough

Walkthrough

The install_biometrics_archiver function now includes an early-exit guard that detects new firmware by checking for the absence of /opt/eight/bin/frank.sh. When the file is missing, the function logs a message and returns without executing the tmpfs mount, timer unit setup, or working-directory patch. Existing behavior is preserved when frank.sh is present.

Changes

Firmware Detection for Biometrics Archiver

Layer / File(s) Summary
Firmware detection early-exit guard
scripts/lib/biometrics-archiver-helpers
A conditional check for /opt/eight/bin/frank.sh is inserted at the start of the installation workflow; if absent, the function logs a skip message and returns successfully without proceeding to tmpfs setup or patching.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~5 minutes

Possibly related PRs

  • sleepypod/core#579: The main PR's change to install_biometrics_archiver in scripts/lib/biometrics-archiver-helpers (early-exit when /opt/eight/bin/frank.sh is missing) directly builds on the retrieved PR's refactor/usage of the same helper for OTA installs and its frank.sh-patching logic.

Suggested labels

released

Poem

A firmware so fresh, with no frank.sh in sight, 🐰
The archiver checks and bows out polite,
No tmpfs to mount, no patch to apply—
Just a gentle return and a skip-safe goodbye! 🎯

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title accurately describes the main change: skipping biometrics-archiver setup on new firmware lacking frank.sh, which directly addresses the Pod 5 firmware issue causing OTA rollbacks.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch worktree-pod5-debug

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@codecov

codecov Bot commented May 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

ng added 4 commits May 17, 2026 14:38
Distinguishes the expected new-firmware path (frank.sh absent + nats-server
active + /persistent/jetstream present) from a broken old-firmware case
(frank.sh deleted, nothing replacing it). frank.sh absence remains the
gating signal — the patch cannot apply without it — but the log line now
gives the operator enough state to diagnose at a glance.
Previously a one-line 'systemctl status sleepypod.service'. Now reports
firmware variant (rat_version, frank.sh presence, frank.service state),
biometrics pipeline (tmpfs RAW vs NATS JetStream — including TCP probe of
4222, jetstream dir size, 'nats stream info raw' check), .RAW counts at
each location, sleepypod modules, biometrics-archiver units, and
firmware-side services. Safe on every firmware variant — every probe
fails silently to 'missing' so the same command works on Pod 3/4/5
regardless of OTA generation.
One-shot Python recorder that subscribes to a NATS subject filter, dumps
every message as NDJSON (subject, headers, payload bytes, CBOR-decoded
shape) for a fixed duration, and prints a per-subject histogram on exit.
Used to learn what frame types Pod 5 new-firmware publishes before
designing the BiometricsSource abstraction in sleepypod-core-56. Bundled
deps via uv shebang; pod-side requires the dac user + the nats-server
already running on localhost:4222 (default on new firmware).
frank.sh was removed from frankenfirmware ~year ago, so its absence is
NOT a reliable canary for the new NATS firmware (April 2026+). Three
firmware variants coexist in the fleet:

  - Old (Pod 3 / Pod 4 / early Pod 5): frank.sh present, .RAW via shim
  - Mid (~2025 → April 2026): frank.sh absent, .RAW direct to /persistent
  - New (April 2026+, e.g. ca35aafa): frank.sh absent, NATS JetStream

The prior gate ('frank.sh absent → skip') caught mid-era pods too, which
would have regressed their archive/prune rotation. Mid-era pods have been
on the warn-and-continue path successfully since PR #588 — preserve that.

Gate strictly on the new-firmware signals: nats-server.service active
AND /persistent/jetstream directory present. Only those pods skip the
helper. sp-status pipeline detection updated to match the 3-state matrix.

ADR 0018 + biometrics-system wiki + scripts/README updated with the
firmware-variant matrix and applicability notes.
@ng ng changed the title fix(sp-update): skip biometrics-archiver setup on new firmware (no frank.sh) fix(sp-update): skip biometrics-archiver setup on new (NATS) firmware + diagnostics May 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant