Harden publication high-water journal before resuming managed preview releases

[rynfar]

Problem

The protected layout-v2 publication high-water journal at 68603ed89bb597cd715fd6a77bc1c39d7e110298 remains the working implementation and its full publication suite passes. However, hosted Pack A/B review found a root-frontier race that prevents us from safely resuming managed immutable preview publication.

Pylon Nightly currently exposes Install/update preview, but no immutable preview publication exists, so the action fails with that exact message. This issue tracks the deferred hardening needed before managed preview/stable artifacts can resume. It does not block manual or existing stable Prime Agent use.

A generation-container layout-v3 (option C) was prototyped and rejected after independent crash/concurrency review. Do not ship or reconstruct that partial patch as-is. Keep the protected v2 behavior until this issue is deliberately resumed.

Safety contract

Cooperative legacy migration

• Require explicit API and CLI acknowledgement that every old v1/v2 process is externally stopped before any migration mutation. Do not claim automatic coexistence with an old process already past projection revalidation.
• Cover the exact historical layouts from the protected implementation, including direct in-place v1 and prior-retired v1 plus regular v2 guard.
• Publish an old-client-visible, impossible-generation blocker bound to the exact pre-block authority, tip, and retirement marker before freezing provenance.
• Preserve original v1 authority directories in place. Freeze both to exact owner read/execute mode, fsync the directory handles and parent, and validate every crash-resume cut.
• Fence v2 with the durable v3 guard before retiring its journal. Derive only from the final path-fenced source.
• Canonical-v3 resume must revalidate final source provenance and repair/verify projection. Concurrent migration helpers must join the same installed result.
• Include an exact protected-old-client test proving a fresh client cannot help or repair projection after the blocker. The external-quiescence precondition remains mandatory for already-running clients.

Generation and rotation authority

• Build a complete hidden generation with an exactly empty epoch and checkpoint-only receipt namespace, fsync it, then use one same-parent rename as linearization.
• Use one exact successor predicate everywhere. Bind epoch increment, deterministic ID and name, predecessor checkpoint/name, immutable tip bytes, history, source provenance, and migration provenance.
• Authenticate the predecessor's exact latest rotation claim/intent/CAS and tip before accepting two finals or retiring a predecessor.
• Fresh preparation must converge a durable two-final crash cut before any callback. A rotation may not return while leaving two finals.
• Bind builder/final, final/retired, retired/deleting, source-retirement, and staging/canonical response-loss joins to the same observed dev/ino. Same-byte replacement is not authority.
• Fully validate successor epoch and receipt semantics before predecessor retirement or reported success.

Typed reads and projection

• Only direct native filesystem evidence may classify a required pinned path as lost. Caller-supplied operation errors and hook errors, including ENOENT, EIO, and EPERM, retain object identity.
• Generic content, stat, inode, or I/O failures must remain terminal even if a rename also occurs.
• If the bounded reader needs to expose an ancestor-rename observation, keep it narrow: exact EOF and bytes, stable opened inode/stat, missing final pathname, unchanged link count, and a private consumer classification after exact digest/schema plus same-inode retired-path proof.
• After every projection rename/readback, rescan the authenticated immutable tip before success. A delayed writer must repair forward rather than report success for stale state.

Receipts, cleanup, bounds, and provenance

• Keep receipt publication order durable: temp+receipt-directory fsync, canonical link+canonical-parent fsync, temp-to-fixed rename+receipt-directory fsync.
• Include receipt temporaries in quiescence. Remove dead or exact decided-claim losers despite PID reuse; keep live unresolved writers as blockers.
• Cap receipt/root/epoch entries and aggregate bytes before nested work. Index canonical entries by inode rather than quadratic receipt scans.
• Size checkpoint bounds for all three base64 state fields and receipt duplication through the supported 16 MiB state limit.
• Port the complete protected v2 grammar: deterministic IDs/history/source/anchor, exact retained predecessor and latest rotation intent, closed epoch sets, indexed winners with valid losers, helpable incomplete states, canonical metadata, and root/entry/byte bounds.
• Commit all exact retained/current v2 epochs and applicable projection bytes into source provenance. Preserve underlying v1 provenance separately.
• Reconstruct and byte-compare the exact state-derived epoch-1 checkpoint for genesis, v1, and v2 migration.

Required evidence before publication resumes

• Keep every still-relevant protected regression matrix.
• Add real hard-process crash cuts before and after every directory fsync/rename boundary.
• Cover builder/final/retired/deleting recovery, root-readdir handoff, every pinned required path, new-inode replacement, generic I/O plus rename, exact historical migration fixtures, concurrent migration, stale projection writers, limits, ownership, and modes on macOS and Linux.
• Run focused and complete publication suites, static checks, and repeated multi-process rotation/migration stress.
• Obtain fresh independent architecture, migration/concurrency, and regression/docs approval of the exact replacement tree.
• Do not publish a managed preview or call a channel stable until those gates pass.

Coordination

• Original delivery tracker: Publish attested preview and stable Pylon Prime releases #29
• Fork umbrella: Track Pylon fork parity and upstream overlap #1
• Pylon Prime umbrella: Prime provider parity: tasks, automation, and upstream watch pylon#114
• Pylon managed distribution work: Verify Pylon Prime distributions without weakening runtime negotiation pylon#193 and Add opt-in side-by-side Pylon Prime install and updates pylon#194