Skip to content

chore: community intake audit, as the satd flavor of bitcoind - #1

Draft
helix-a wants to merge 5 commits into
mainfrom
chore/community-audit
Draft

helix-a wants to merge 5 commits into
mainfrom
chore/community-audit

Conversation

@helix-a

@helix-a helix-a commented Sep 20, 2026 •

Copy link
Copy Markdown
Member

Start9's intake review of the submission. Draft, pending StartOS flavor-switch verification. satd 0.6.1 (merged via #3) contains the upstream reindex fixes, and #4 updates the package to SDK 3.0.6. Nothing here merges until Core → satd → Core is re-verified on StartOS, including shutdown and resumption during reindex.

Why a flavor of bitcoind

satd is a Bitcoin node implementation, so on StartOS it belongs where Core and Knots are: as a flavor of the bitcoind package (id: 'bitcoind', version #satd:0.6.1:0). That does two things a standalone package can't:

  • Mutual exclusion. One bitcoind at a time, so satd can only be installed instead of Core or Knots — never beside them with a second full chain and a second set of indices on the disk.
  • Dependents refuse it unless they opt in. A flavored version never matches an unflavored range, so Lightning, electrs, Mempool and the rest see Bitcoin as unsatisfied while satd is installed. Since satd publishes no rawblock/rawtx ZMQ, that's the truth; a dependent that has verified itself against satd can add || #satd:>=0.6.1:0 later.

The repo keeps its name; the marketplace title is Bitcoin (satd).

The flavor switch

migrations.other, keyed by Core's 28–31 series and #knotsprerdts, carries the switch both ways over the shared blocks/. satd reads Core's blk*.dat/rev*.dat layout — xor.dat included — so the block files are never touched; each side rebuilds its own chainstate and indices from them:

  • Core → satd (up) removes Core's chainstate/, chainstate_background/, indexes/, peers.dat, mempool.dat and arms a one-shot --reindex in the store.
  • satd → Core (down) removes satd's state and rendered config and sets reindexBlockchain: true in Core's store.json, so Core reindexes from the same files and comes back with a default mainnet configuration.

Both directions clean implementation-specific network state at the root and in signet, testnet4, testnet3 and regtest directories, preserving each network's block files. A flavor switch does not preserve the selected test network.

Verified (Core 31.0 signet datadir, 322,806 blocks, 22 GB): satd indexed all 322,807 headers from Core's files, selected Core's exact tip, and replayed with full validation and no download; every Core-written file was byte-identical afterwards. Not verified: the down direction, and the switch on a StartOS box — the only box available runs the Core node its Lightning stack depends on. Please run it on a non-production StartOS box with 0.6.1.

Original upstream findings (fixed in satd 0.6.1; runtime re-test pending)

The following observations came from the original 0.5.2 signet test. epochbtc/satd#926 addresses interrupted reindex and SIGTERM handling; epochbtc/satd#934 addresses duplicate genesis writes. The remaining throughput concern is not independently blocking.

Found while running the original switch on signet:

  1. A killed reindex does not resume from the block files. After a SIGKILL at block 231,797 of 322,806, satd restarted with no memory of the 91k blocks still on disk, went back to network IBD from its flushed tip (~600 MB downloaded in three minutes), and wrote the re-downloaded blocks into new blk00149+.dat files beside the originals — appending into Core's last file too. On mainnet that is hundreds of gigabytes re-downloaded and stored twice, which defeats the flavor switch.
  2. SIGTERM is ignored while connecting blocks. docker stop with a 600 s grace period ran the full 600 s and killed it. StartOS sends SIGTERM and kills after the package's sigtermTimeout, so any stop, reboot or settings change during the reindex produces chore: community intake audit, as the satd flavor of bitcoind #1. Seen on the StartOS box too, on an ordinary sync.
  3. Reindex throughput. The first 220k signet blocks replayed in ~20 minutes; the recent large ones went at ~210 blocks/min at default dbcache, using 1.5 cores of 12 — latency-bound, not CPU-bound. Extrapolated to mainnet, a Core→satd switch is days. Not a blocker on its own, but it makes chore: community intake audit, as the satd flavor of bitcoind #1 and fix: run satd with RUST_LOG=info #2 hard to avoid.

Reproduction is a Core-synced signet datadir with chainstate/ removed, then satd --chain=signet --reindex; stop it mid-replay.

The rest of the audit

Beyond the flavor conversion, all verified on a StartOS 0.4 box as the standalone package before the conversion (install, mainnet first start, Network → signet with the peer port following, every interface serving through the OS proxy, every action, stop/start, backup/restore round-trip, clean reinstall):

  • RPC interface was unusable off the box — cookie auth only. Set RPC Credentials generates a Core-format rpcauth line (username satd, hash in the store, password shown once) passed on satd's command line. Verified 401 without, 200 with, on the bridge and the LAN TLS port.
  • MCP Token was a "show" action on a token satd-init minted, with no rotation. Now Set MCP Token: the package seeds a token at install and writes secrets/mcp-token on every start from the store; the action rotates it. Verified 401 → 200, and 403 for a Host not in MCP Hostnames.
  • MCP binds as https so its address carries a scheme (it was protocol: null, rendering as a bare host:port, and masked with nothing secret in the URL); Electrum gets ssl:///tcp:// schemes like electrs; action prefills return only their own fields rather than the whole store (which now holds the token).
  • The Blockchain Sync check names the network ("Syncing Signet blocks: …") and passes counts as strings — start-sdk 2.0.9's i18n() throws on a number under the container's LANG=C.UTF-8 (i18n() throws RangeError on a numeric or Date parameter — LANG=C.UTF-8 in service containers is not a BCP-47 tag (projects/start-sdk/lib/i18n/index.ts) Start9Labs/start-technologies#3839, fixed on master, not yet released).
  • packageRepo → this fork; image pinned by tag alone; every string, the description and release notes translated into es/de/pl/fr (the translations file was an empty stub); workflows replaced with the fleet's set on main; AGENTS.md, CLAUDE.md, .dockerignore, UPDATING.md added; template .gitignore/Makefile/overrides; icon re-encoded from 195 KiB to 20 KiB; README.md and instructions.md rewritten to the packaging guide's shape. Your test/ suite is kept and updated to the edited sources.

Review follow-up verification

  • Applied the reviewer's shared network-file cleanup proposal and documented the mainnet reset in both guides; normalized README table formatting.
  • SDK 3.0.6 is the current npm release and already landed with its dependency/plumbing update in Update start-sdk to 3.0.6 #4.
  • Passed npm run check (typecheck and all 11 tests), Prettier, and make bitcoind.s9pk through helix-build. Inspected the returned artifact: bitcoind, #satd:0.6.1:0, SDK 3.0.6.
  • A temporary filesystem harness invoked both production migration bodies with mocked SDK/store boundaries: all five network directories cleaned, block/rev/xor bytes preserved, reindex flags set, repeated calls succeeded. This is not a StartOS integration test.
  • GitHub's build job remains skipped by the draft guard. A successful local build is not a successful CI run. Keep draft until the StartOS round-trip and interrupted-reindex tests are completed; no new runtime verification is claimed.

From here on this fork is the upstream for the package: later versions come as pull requests against it.

🤖 Generated with Claude Code

Helix-Harness: pi
Helix-Model: openai-codex/gpt-6.1-sol

Make the package a flavor of bitcoind (id bitcoind, version #satd:0.5.2:0)
so it installs in place of Core or Knots rather than beside them, and so no
dependent's unflavored bitcoind range matches it: satd publishes no
rawblock/rawtx ZMQ, so Lightning, electrs and Mempool cannot use it and
should say so. migrations.other keyed by Core's 28–31 series and the Knots
flavor carries the switch both ways over the shared blocks/ directory: up
removes Core's chainstate and indexes and arms a one-shot --reindex in the
store; down removes satd's state and sets reindexBlockchain in Core's
store.json. Up was verified on a Core-synced signet datadir (headers indexed
from Core's xor'd block files, Core's tip chosen, replay validated with no
download); down has not been run.

Own the credentials the way the guide asks: Set RPC Credentials generates
a Core-format rpcauth line passed on satd's command line (the interface was
cookie-only, unusable off the box), and Set MCP Token rotates a token the
package writes to secrets/mcp-token on every start instead of showing one
satd-init minted. MCP binds as https so its address carries a scheme, Electrum
gets ssl:// and tcp:// schemes, action prefills return only their own fields,
and the sync health check names the network and passes counts as strings
around start-sdk 2.0.9's i18n locale bug (start-technologies#3839).

Point packageRepo at the Start9-Community fork, pin the image by tag alone,
translate every string and the manifest into the fleet's four locales, replace
the workflows with the fleet's set on main, add AGENTS.md, CLAUDE.md,
.dockerignore and UPDATING.md, re-encode the icon under 40 KiB, and rewrite
README.md and instructions.md to the guide's shape.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
StartOS starts every daemon with RUST_LOG=warn,start_core=debug in its
environment, and satd reads RUST_LOG in preference to its own default of
info. The service log carried warnings only: no sync progress, no
reindex phases, no shutdown steps. The satd daemon now sets
RUST_LOG=info, and a test checks it does.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@MattDHill
MattDHill marked this pull request as ready for review October 9, 2026 22:04
@helix-a

helix-a commented Oct 9, 2026

Copy link
Copy Markdown
Member Author

Cross-review requested.

@MattDHill
MattDHill marked this pull request as draft October 9, 2026 22:11
@MattDHill

MattDHill commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

@bkeroack thanks for landing the reindex fixes. epochbtc/satd#926 covers both blockers above (epochbtc/satd#906, epochbtc/satd#907), and epochbtc/satd#934 takes care of the duplicate genesis block.

Both merged after v0.6.0 was tagged, so no satd release has them yet. When you cut one, please open a PR against chore/community-audit that bumps the image to it, the same way you did #2. We'll then re-test switching from Core to satd and back on a StartOS box and take this out of draft.

@helix-a

helix-a commented Oct 9, 2026

Copy link
Copy Markdown
Member Author

Keeping this draft pending the release-bump PR against chore/community-audit. Once that lands, the acceptance check is Core → satd → Core on StartOS, including interrupting and restarting satd during reindex to verify shutdown and resumption before marking this ready.

@helix-a helix-a left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested on 7b78382c19f22e4dcf5b0696cb78a50693816d80.

  1. Clean implementation-specific state in every network directory during flavor switches. startos/versions/current.ts only removes root-level chainstate/peer/mempool files, although satd supports signet, testnet4, testnet3 and regtest data beneath separate directories. Those directories retain the outgoing implementation's state, rather than undergoing the conversion this migration promises. The proposal shares the network-file removal primitive between both directions, preserves every network's block files, and updates both guides to make the existing mainnet reset explicit. This does not claim to preserve a test-network selection.
  2. Update the SDK and its dependency plumbing. The package pins 2.0.9; npm's latest is 3.0.4. I tested the exact bump and stabilized its lockfile, but typechecking fails at removed setupDependencies, the removed manifest dependencies field, and the now-required third buildManifest argument. A complete update needs the new Dependencies API and refreshed plumbing. I reverted that incomplete bump rather than publish a broken proposal: startos/index.ts is explicitly protected by the workspace's DO NOT EDIT rule, so its refresh needs an authorized plumbing update. This remains required before approval.
  3. Obtain a successful PR build. The sole check for the reviewed head is build: SKIPPED, not a successful build. Also, the PR is ready for review while its description still explicitly says it is a draft blocked on upstream interrupted-reindex/SIGTERM fixes and unverified flavor-switch behavior. Keep it draft until those blockers are resolved; local checks do not establish that the switch works on StartOS.

Verified the PR merges with origin/main. On the migration proposal with the original SDK: npm ci, npm run check (typecheck and all 11 tests), Prettier, and make bitcoind.s9pk passed through helix-build. Inspected the returned package's manifest and commitment. A temporary filesystem harness invoked both production migration bodies with mocked SDK/store boundaries: all five network directories were cleaned, block/xor bytes were preserved, reindex flags were set, and repeated calls succeeded. No StartOS runtime or full blockchain switch was run; the existing service verification is acknowledged, not extended by these checks.

Proposed changes: 7b78382...proposal/1

Helix-Harness: pi
Helix-Model: openai-codex/gpt-6.1-sol

This was referenced Oct 15, 2026
bkeroack and others added 3 commits October 16, 2026 19:16
* Update start-sdk to 3.0.6

Follows the 3.0 migration in the SDK changelog and matches the 3.0.6
package template:

- Dependencies are declared once in dependencies.ts with
  sdk.Dependencies.of() (satd has none) and passed to buildManifest and
  setupInit; the manifest's dependencies field is gone.
- Action results set launchable, now required on every single value.
- The store's file model uses z.looseObject, as 3.0 asks of every file
  model.
- The SDK supplies TypeScript, Prettier and ncc: devDependencies, the
  build and prettier scripts and the prettier block are dropped for
  .prettierrc. `npm run check` stays, and the Makefile's TS_CHECK runs it,
  so the build gate runs test/ as well as tsc.

The built manifest differs from the 2.0.9 build only in sdkVersion,
osVersion (0.4.0-beta.10 to 0.4.0.2, which 3.0 requires) and
emulateMissingAs, which 3.0 replaced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Finish the package-lock.json fixed point after the SDK bump

A second npm install records the SDK's bundled @start9labs/start-core
tree; a third leaves the lock unchanged, as the packaging guide's
"Bumping the SDK" procedure requires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* satd 0.6.1

Pins ghcr.io/epochbtc/satd:0.6.1, the first release with the reindex
fixes this draft waits on: a stopped or killed reindex continues from
the block files on disk, SIGTERM stops a block replay, and a full
reindex no longer appends a genesis block to Core's last block file.

test/upstream/satd-init is vendored from v0.6.1. satd-init stopped
reading SATD_STACK_SUBNET in 0.6.0 (the template no longer renders a
subnet into rpcallowip), so the package no longer passes it. The README
and instructions drop the advice not to interrupt the switch, and the
limitation it described becomes the one still open: reindex speed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Release notes: warn that a 0.5.x datadir rebuilds its chainstate

The update crosses 0.6.0's chainstate schema change (3 to 7), and satd-init
sets upgradechainstate=1, so an existing 0.5.x datadir rebuilds its
chainstate and indexes from the local block files on the first start. Say so
in every locale, link the upstream changes, and list the metrics TLS
listener (9336) among the unexported ones.

Helix-Harness: pi
Helix-Model: openai-codex/gpt-6.1-sol

* Update start-sdk to 3.0.6

Follows the 3.0 migration in the SDK changelog and matches the 3.0.6
package template:

- Dependencies are declared once in dependencies.ts with
  sdk.Dependencies.of() (satd has none) and passed to buildManifest and
  setupInit; the manifest's dependencies field is gone.
- Action results set launchable, now required on every single value.
- The store's file model uses z.looseObject, as 3.0 asks of every file
  model.
- The SDK supplies TypeScript, Prettier and ncc: devDependencies, the
  build and prettier scripts and the prettier block are dropped for
  .prettierrc. `npm run check` stays, and the Makefile's TS_CHECK runs it,
  so the build gate runs test/ as well as tsc.

The built manifest differs from the 2.0.9 build only in sdkVersion,
osVersion (0.4.0-beta.10 to 0.4.0.2, which 3.0 requires) and
emulateMissingAs, which 3.0 replaced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Finish the package-lock.json fixed point after the SDK bump

A second npm install records the SDK's bundled @start9labs/start-core
tree; a third leaves the lock unchanged, as the packaging guide's
"Bumping the SDK" procedure requires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: helix-a <267227783+helix-a@users.noreply.github.com>
Helix-Harness: pi
Helix-Model: openai-codex/gpt-6.1-sol
@helix-a

helix-a commented Oct 17, 2026

Copy link
Copy Markdown
Member Author

Addressed the code feedback in 98c5884:

  1. Applied the network cleanup from proposal/1 to both migration directions, including Core’s background chainstate. Block files remain untouched in every network directory. Both guides now explain the mainnet reset.
  2. Update start-sdk to 3.0.6 #4 already landed SDK 3.0.6 and the dependency/plumbing migration; the branch matches the current npm release. No further protected-file changes were needed.
  3. helix-build passed typechecking, all 11 tests, formatting, and make bitcoind.s9pk. Inspected the returned manifest and commitment. A temporary harness exercised the production migration bodies with mocked SDK/store boundaries: five networks cleaned, block/rev/xor bytes preserved, reindex flags set, and repeated calls succeeded.

The PR remains draft. #3 has now landed satd 0.6.1, but Core → satd → Core on StartOS, including interrupted-reindex recovery, is still unverified. GitHub CI is skipped by the draft guard; the successful local build does not resolve that CI requirement. Updated the description to distinguish resolved upstream/package work from these remaining acceptance checks.

@helix-a

helix-a commented Oct 17, 2026

Copy link
Copy Markdown
Member Author

Please re-review the migration changes and existing SDK 3.0.6 update. The PR remains draft pending the StartOS acceptance tests and a successful GitHub build; neither is claimed complete.

@helix-a

helix-a commented Oct 17, 2026 •

Copy link
Copy Markdown
Member Author

Checked both CI notifications against GitHub: the run on 7a90f25 and the current head’s sole build check on 98c5884 both concluded skipped, not success. Rechecked run 38990421700 after its own notification: still skipped, no steps, and no failed logs. No failed job needs fixing, and no successful GitHub build is established. The draft/runtime acceptance gates remain unchanged. Reported the notification mismatch in Start9Labs/helix-startos-internal#25.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants