Skip to content

chore(sp-bundle-logs): capture streamer attach log and archive/disk pressure#660

Merged
ng merged 1 commit into
devfrom
chore/sp-bundle-logs-streamer-retention
Jul 12, 2026
Merged

chore(sp-bundle-logs): capture streamer attach log and archive/disk pressure#660
ng merged 1 commit into
devfrom
chore/sp-bundle-logs-streamer-retention

Conversation

@ng

@ng ng commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Why

sp-bundle-logs already captures the RAW routing (where frames land, where each reader looks, tmpfs state), but two live field reports needed signals it didn't collect:

  1. "Only DEV shows in the Sensors tab" (fix(streaming): skip SEQNO and prefer biometrics RAW #608) — caused by the streamer tailing SEQNO.RAW (the 16-byte counter) instead of a real capture. The tell is in sleepypod.service's log: [sensorStream] Switched to RAW file: SEQNO.RAW followed by Resync: spam. Not previously in the bundle.
  2. "Pod stopped working, /persistent full, update fails with No space left on device" — a stalled biometrics pruner letting the gzip archive grow unbounded. The bundle recorded the pruner's journal but not the archive size vs. disk free, so the smoking gun wasn't obvious.

Change

Two probes appended to the existing raw/ section:

  • raw/ws-3001.txt + raw/sensorstream-log.txt — is the streamer WS server up, and what did it last attach to (+ any resync spam).
  • raw/archive-size.txtdu -sh of the cold archive, gz file count, and df -h /persistent side by side.

No new dependencies; all best-effort under the existing capture helper. Pruner/archiver journals continue to come from the journals/ section.

Test plan

  • bash -n clean.
  • Nested-quote bash -c snippets smoke-tested locally (archive-size probe emits count + df; sensorStream grep is a no-op on a non-pod).
  • On-pod: run sp-bundle-logs, confirm raw/ws-3001.txt, raw/sensorstream-log.txt, and raw/archive-size.txt are populated.
🛏️ Beam this PR onto a sleepypod

Pick the snippet that matches your pod. Already-installed pods can't use the curl bootstraps below — sleepypod's egress firewall DROPs github.com, and the script that knows how to unblock WAN is the very file curl is trying to fetch. Use sp-update instead; it's on disk and opens WAN as its first step.

🔄 Already on a sleepypod (most common — review a PR on a running pod):

sudo sp-update chore/sp-bundle-logs-streamer-retention

🚀 Fresh pod / first install — bootstraps from GitHub, rebuilds every push:

curl -fsSL https://raw.githubusercontent.com/sleepypod/core/chore/sp-bundle-logs-streamer-retention/scripts/install \
  | sudo bash -s -- --branch chore/sp-bundle-logs-streamer-retention

🎯 Pin to this exact build (run 28831420964 · 71859db) — fresh-pod path, locked to one CI artifact for reproducible review:

curl -fsSL https://raw.githubusercontent.com/sleepypod/core/71859db98d8fdfd0f64cdf49f8a00b089fe418c0/scripts/install \
  | sudo bash -s -- \
      --branch chore/sp-bundle-logs-streamer-retention \
      --artifact-url 'https://nightly.link/sleepypod/core/actions/runs/28831420964/sleepypod-core.zip'

🧊 Artifact: sleepypod-core · self-destructs in 30 days 💥

Summary by CodeRabbit

  • New Features

    • Added more diagnostic and storage details to the bundled logs, including network, service journal, disk usage, and archive size information.
  • Bug Fixes

    • Improved service setup so it only uses available system groups, reducing startup failures on systems where expected groups are missing.

…ressure

The raw section already records where frames land and where each reader looks,
but not the two signals behind the most recent field reports:

- the WebSocket streamer's attach log ("Switched to RAW file: …" / "Resync:"),
  which pinpoints the wrong-file bug where the streamer tails SEQNO.RAW and only
  DEV shows in the Sensors tab
- gzip archive size vs /persistent free space, which reveals a stalled pruner
  before it fills the disk and blocks updates

Both are diagnosable from the bundle alone now instead of asking the user to
hand-run probes.
@coderabbitai

coderabbitai Bot commented Jul 5, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

This PR adds new diagnostic captures to sp-bundle-logs (port 3001 status, sensorStream journal logs, biometrics-archive size/gzip count, disk usage) and reworks scripts/install to dynamically compute SupplementaryGroups (dac, systemd-journal or adm fallback) for the sleepypod.service unit, only emitting the line when groups exist.

Changes

Bundle Log Diagnostics

Layer / File(s) Summary
Raw frame routing and retention diagnostics
scripts/bin/sp-bundle-logs
Adds checks for port 3001 listening status, tails last 40 sensorStream journal lines, and captures biometrics-archive size, gzip file count, and disk usage, all written under raw/.

Systemd Service Group Configuration

Layer / File(s) Summary
Dynamic SupplementaryGroups computation
scripts/install
Builds a SUPP_GROUPS array conditionally including dac and systemd-journal (or adm fallback), emitting the SupplementaryGroups line only when the list is non-empty.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • sleepypod/core#115: Both PRs modify the sleepypod.service systemd unit generation logic in scripts/install.
  • sleepypod/core#580: Both PRs extend the sp-bundle-logs diagnostic bundle script with additional captured diagnostics.
  • sleepypod/core#629: Both PRs add new raw/-staged diagnostic captures to sp-bundle-logs.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main bundle-logging change and its archive/disk-pressure diagnostics.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch chore/sp-bundle-logs-streamer-retention

Warning

Review ran into problems

🔥 Problems

Git: Failed to clone repository. Please run the @coderabbitai full review command to re-trigger a full review. If the issue persists, set path_filters to include or exclude specific files.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Jul 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
scripts/bin/sp-bundle-logs (2)

172-173: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider ss -ltnp for process attribution.

ss -ltn confirms the port is listening but not which process/PID owns it. Since the goal is diagnosing whether the "right" streamer attached, including -p would directly show the owning process for cross-checking against the sensorStream journal tail.

♻️ Optional enhancement
-capture "raw/ws-3001.txt"          bash -c 'ss -ltn 2>/dev/null | grep -q ":3001 " && echo "port 3001 listening" || echo "port 3001 NOT listening"'
+capture "raw/ws-3001.txt"          bash -c 'ss -ltnp 2>/dev/null | grep ":3001 " || echo "port 3001 NOT listening"'
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/bin/sp-bundle-logs` around lines 172 - 173, The port-check captured
by the `capture` call for `raw/ws-3001.txt` only verifies that port 3001 is
listening, but it does not identify the owning process. Update the `ss`
invocation in this log bundle step to include process attribution so the output
can be cross-checked against the `sensorStream` journal tail; keep the existing
`capture` structure and related `journalctl` capture unchanged.

173-173: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value

journalctl -b only covers the current boot.

If the sleepypod service (or the pod) restarted after the misattachment happened but before the bundle is collected, the relevant [sensorStream] attach log will already be outside the current boot window and this capture will come back empty. Worth confirming this is intentional given the bundle is meant to catch a currently-reproducing issue, or whether recent-boots-inclusive (-b -1 or a --since window) would be more robust.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/bin/sp-bundle-logs` at line 173, The current `capture` for
`raw/sensorstream-log.txt` only reads `journalctl` from the current boot via
`-b`, so it can miss the relevant `[sensorStream]` attach log after a restart.
Update the `journalctl` invocation in `sp-bundle-logs` to include recent history
as intended, such as a recent-boots window or a `--since`-based range, while
keeping the existing `grep -iE "[sensorStream]"` and `tail -40` filtering in
place.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@scripts/bin/sp-bundle-logs`:
- Around line 172-173: The port-check captured by the `capture` call for
`raw/ws-3001.txt` only verifies that port 3001 is listening, but it does not
identify the owning process. Update the `ss` invocation in this log bundle step
to include process attribution so the output can be cross-checked against the
`sensorStream` journal tail; keep the existing `capture` structure and related
`journalctl` capture unchanged.
- Line 173: The current `capture` for `raw/sensorstream-log.txt` only reads
`journalctl` from the current boot via `-b`, so it can miss the relevant
`[sensorStream]` attach log after a restart. Update the `journalctl` invocation
in `sp-bundle-logs` to include recent history as intended, such as a
recent-boots window or a `--since`-based range, while keeping the existing `grep
-iE "[sensorStream]"` and `tail -40` filtering in place.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 0e16c0d0-4a6c-4a3b-af5d-7bb867c5929d

📥 Commits

Reviewing files that changed from the base of the PR and between 7593463 and 879a892.

📒 Files selected for processing (2)
  • scripts/bin/sp-bundle-logs
  • scripts/install

@ng
ng force-pushed the chore/sp-bundle-logs-streamer-retention branch from 879a892 to 71859db Compare July 6, 2026 23:54
@ng
ng merged commit 821ac78 into dev Jul 12, 2026
14 checks passed
@ng
ng deleted the chore/sp-bundle-logs-streamer-retention branch July 12, 2026 00:43
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 2.5.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant