Phase 6: v1 → v2 SDK migration (uxf-v2 fork review artifact)#640
Draft
vrogojin wants to merge 1260 commits into
Draft
Phase 6: v1 → v2 SDK migration (uxf-v2 fork review artifact)#640vrogojin wants to merge 1260 commits into
vrogojin wants to merge 1260 commits into
Conversation
…test-coverage test(payments/transfer): close V6-RECOVER test-coverage gap + nightly soak CI (Audit #333 follow-up)
…aths Created in response to #363 — the post-mortem of #360. The lesson from #360: do not propose perf fixes from static analysis. Measure first, fix second. This commit adds the missing measurement step that #360 should have started with. * core/perf-counters.ts (new) — minimal counter / timer module. - `incr(name)`, `observeMs(name, ms)`, `time(name, fn)` API. - Gated by SPHERE_PERF=1 env var (or localStorage.SPHERE_PERF=1 in the browser). When off, every entry point is a single boolean check + early return. - When on, dumps a snapshot of all counters every SPHERE_PERF_DUMP_MS (default 5000ms) via `logger.info('perf', …)`. - 15 unit tests covering the on/off semantics, clamp behaviour, and throw-still-records contract. * Wired at the hot paths the #360 findings claimed but never measured: - OrbitDbAdapter.putEntry — wall-clock + count, split bundle-ref keys from everything else. Directly answers the maintainer's batching hypothesis (is every CID write a full round-trip?). - OrbitDbAdapter.onReplication handler — fires/sec + callback ms. Tests Finding #1's "every local write fires a full re-sync" claim. - BundleIndex.listBundles — call rate + db.all() walk ms. Tests Finding #3's "5× per flush" claim. - ipfs-client.fetchFromIpfs — split local-Helia hits from HTTP gateway fetches so the two cost models are separable. - ipfs-client.fetchCarFromIpfs — per-CAR walk wall-clock + blocks + bytes. - ipfs-client.pinCarBlocksToIpfs — pin wall-clock + blocks + bytes. - flush-scheduler.flushToIpfs — call rate + total ms (via try/finally body extraction to avoid touching every early-return). - UxfPackage.computeVerifiedProofs — calls + per-verifier RPC ms + ok/threw split. Tests Finding #2's O(B²·P) claim. * No behaviour change. All 3352 tests in tests/unit/profile/, tests/unit/uxf/, tests/unit/core/ pass. Type-check + lint clean. Usage: SPHERE_PERF=1 SPHERE_DEBUG=perf=info bash .tmp/soak-264.sh Every 5s the daemon emits a counter snapshot. After the soak, post- process the log to build a flame-shaped table of what actually dominates the wall-clock. This is NOT a fix. It is the instrumentation that should have preceded any #360 work. See #363 for the post-mortem.
… paths; add SPHERE_SKIP_RULE4 prototype Follow-up to c5962f0 (#363). The first round of counters surfaced a ~213 s gap between `uxf.computeVerifiedProofs` outer-wall and its inner verify-call total. This commit extends the measurement to (a) every aggregator HTTP entry point, (b) the §D.4 cross-device recovery path, and (c) the apply-snapshot blob-fetch / parse / dispatch sub-steps. It also adds an env-gated bypass of the O(N²) Rule-4 pairwise UXF verification loop in `load()` to A/B-measure its real wall-clock contribution. Files + counters added: * oracle/UnicityAggregatorProvider.ts - `aggregator.rpc.<method>{,.http_error,.rpc_error}` wraps every JSON-RPC call by method name (catches every getProof, isSpent, submit, mint, validateToken etc. that goes through rpcCall). - Per-public-method outer counters via `aggregator.<method>` for submitCommitment, submitMintCommitment, getProof, waitForProof, waitForProofSdk, validateToken, verifyInclusionProof, isSpent, getTokenState, getCurrentRound, mint. Each wraps the existing body via a private `_<method>Impl` so the SDK-client path is captured separately from rpcCall HTTP. * profile/profile-token-storage-provider.ts - `profile.load.{calls,totalMs}` on the public `load()` entry. - `profile.load.rule4Pairwise{Invocations,Ms,Threw}` measures the O(N²) `computeVerifiedProofs` loop independently from the rest of load (lines 1521-1533). - `profile.load.rule4Skipped` increments when SPHERE_SKIP_RULE4=1. - SPHERE_SKIP_RULE4=1 env flag bypasses the pairwise loop entirely (mirrors the existing graceful-degradation fallback the codebase already takes on verifier failure). For A/B measurement only; the design-correct replacement is a snapshot-source detection gate, tracked in the follow-up issue. - `profile.applySnapshot{,.fired}` on applySnapshotIfWired. * profile/aggregator-pointer/ProfilePointerLayer.ts - `pointerLayer.recoverLatest`, `.discoverLatestVersion`, `.probe`, `.classify` — recovery-time pointer-scan + per-version probe entry points. * profile/aggregator-pointer/discover-algorithm.ts - `pointerLayer.findLatestValidVersion` — the actual walk-and- classify algorithm. * profile/profile-snapshot-cache.ts - `snapshotCache.readSnapshot`, `.writeSnapshot`. * profile/factory.ts - `applySnapshotCb.{fetchRootMs,parseSnapshotMs,dispatchMs,totalMs}` decomposes the snapshot-apply callback registered by setApplySnapshotCallback. Validates the "scan → fetch-blob → rebuild" design: fetchRoot avg 6 ms, parse 5.5 ms, dispatch 385 ms — total ~400 ms per apply. All counters gated by SPHERE_PERF=1. Zero overhead when off (one boolean check + early return). ## A/B result (single iteration of manual-test-full-recovery.sh) Baseline (Rule-4 enabled, full instrumentation): - Total wall: 622 s - §D.4 recovery: 229 s - `profile.load.rule4PairwiseMs`: 222.5 s (35.8% of soak) - `profile.load.totalMs`: 239 s (38.4% of soak) Treatment (SPHERE_SKIP_RULE4=1): - Total wall: 413 s (-209 s / -33.6%) - §D.4 recovery: 123 s (-106 s / -46%) - `profile.load.rule4PairwiseMs`: 0 (bypassed) - `profile.load.totalMs`: 27.7 s (-88%) Two hypotheses rejected by the new counters: - Aggregator HTTP is the bottleneck: total 1.97% of soak (11.5 s of 622 s). `aggregator.verifyInclusionProof` count exactly matches `uxf.computeVerifiedProofs.verifyMs` count (565 171 in baseline), so the inner verification is local crypto on cached proofs, not a round-trip to L3. - `OrbitDbAdapter.putEntry` bundle-key batching wins: bundle puts fire 0.13/sec, total 1.1% of soak — too infrequent to matter. Hypothesis VALIDATED: "convert UXF → TXF in memory, verify at TXF level not UXF level" (the maintainer's design intuition). The Rule-4 pairwise loop is 35.8% of the soak; bypassing it for snapshot-sourced loads recovers the 213 s gap directly. ## Open work tracked in the follow-up issue - pointerLayer.recoverLatest fires 108× per single recovery (design expects 1) - profile.applySnapshot fires 23× during `sphere clear` (architectural smell) - 47 s of §D.4 still uninstrumented (likely token deserialization + OrbitDB dispatch internals) - Convert SPHERE_SKIP_RULE4 prototype to design-correct snapshot- source gate - Kubo Swarm.ConnMgr cap A/B (operational, no SDK change) - Fix §D.5 snapshot-diff contamination from SPHERE_PERF=1 multi-line dumps Full report at /tmp/soak-metrics/issue-363/REPORT.md (497 lines) with all raw soak artifacts preserved alongside.
…in snapshot diff The §D.5 assertions in manual-test-full-recovery.sh compare normalized snapshots before/after the IPFS-only recovery. When `SPHERE_PERF=1` is set (e.g. while collecting #363 instrumentation data), the perf-counter auto-dump fires on a setInterval and `console.log`s a multi-line block: [perf] [perf-counters] snapshot: { 'profile.applySnapshot': { count: 42, totalMs: 123.4, ... }, 'aggregator.fetch': { count: 7, totalMs: 12.3, ... } } The previous per-line sed filter stripped the timestamped *opening* line (via the ISO sed rule) but left the continuation rows and the bare closing `}` at column 0 untouched, producing spurious diffs between snapshots that should otherwise compare equal. Result: every SPHERE_PERF=1 soak failed at §D.5 even when the wallet code was correct. Fix (issue-364 option a): prepend an awk state-machine pass to normalize_snapshot() that detects the opening `[perf-counters] snapshot:` substring (in both timestamped and untimestamped forms), classifies the line as single-line (ends with `}`) or multi-line (ends with `{`), and drops every line in the block through the matching column-0 `}` inclusive. Runs BEFORE the existing ISO sed rule so the timestamped opening line is detected before it's eaten. The awk pass relies on a structural invariant of Node's util.inspect: the top-level `}` is always at column 0 regardless of inner brace nesting. CLI commands captured by the harness (`balance`, `tokens`, `invoice list`, `status`, `invoice status`) all emit plain text — none emit a bare `}` at column 0 — so the state machine cannot false- positive on legitimate output. Also adds a self-test stanza guarded by `RUN_NORMALIZE_TESTS=1` that runs 7 normalize_snapshot() cases (multi-line, single-line, timestamped opening, multiple consecutive blocks, legacy strips intact, defensive `}` inside counter name, identity passthrough). The gate bypasses the teardown trap so no workspace is created. Verification: - `bash -n` clean - `shellcheck` clean (no new warnings; pre-existing SC2034 on `banner()`'s unused `now` local is unchanged) - `RUN_NORMALIZE_TESTS=1 bash manual-test-full-recovery.sh` — all 7 self-tests pass Out of scope for this commit: option (b) — routing the perf dump to stderr in core/perf-counters.ts. The awk filter is sufficient and keeps the runtime behaviour unchanged, which matters for operators who want the perf snapshots interleaved on stdout for grep-friendly post-mortems. Refs: GH issue #364 Item #6
…clear Issue #363 baseline soak recorded 23 `profile.applySnapshot` invocations out of 80 total during the 41 s §D.3 phase (`sphere clear on all wallets`). A clear command applying snapshots is structurally wrong — clear is supposed to delete state, not seed it — and the IPFS round-trips driven by those applies were a measurable contributor to the 41 s wall. ## Trace `ProfileTokenStorageProvider.clear()` awaits `db.all()` + per-key `db.del()` on a connected provider. While the await chain is in flight, the periodic pointer-poll (`LifecycleManager.runPointerPollOnce`) can fire: `pointer.recoverLatest()` returns a snapshot CID, the closure calls `host.applySnapshotIfWired(cid)`, the wired callback fetches the CAR from IPFS, parses the lean snapshot, and dispatches per-writer JOIN against state that is about to be wiped by the surrounding `clear()`. The existing `isShuttingDown`/`hasShutdown` gate in `_applySnapshotIfWiredImpl` only covers the destroy-path shutdown sequence — `clear()` on a live (non-shutdown) provider has no gate. The 23 fires were precisely these clear-time races. ## Fix (strategy a) - Add a private `isClearing` latch on `ProfileTokenStorageProvider`, set at the top of `clear()`, unset in `finally`. - Extend `_applySnapshotIfWiredImpl` to check the latch first; when set, bump `profile.applySnapshot.suppressedDuringClear` and return null without invoking the callback. - The pointer-poll loop, its caller (`recoverFromAggregatorPointerBestEffort`), and any host wire-up code all observe the suppression transparently via the existing `null` return contract — no signature changes. Strategy (b) (stop the periodic poll first inside `clear()`) was considered but adds cross-module coupling: the poll timer is owned by `LifecycleManager` and has no symmetric pause API. Strategy (a) guarantees suppression at the dispatch boundary regardless of which caller raced, with a single-line check on the hot path. ## Tests `tests/unit/profile/profile-token-storage-clear-suppress-apply-snapshot-364.test.ts` (6 cases): - Mid-`clear()` race: spy on `db.all` to fire `applySnapshotIfWired` before returning — asserts the spy returns null and the wired callback is never invoked. - Counter assertion: two mid-clear races bump `profile.applySnapshot.suppressedDuringClear` to 2 with `profile.applySnapshot.fired` staying at 0. - Post-`clear()` normal operation: a fresh `applySnapshotIfWired` after `clear()` returns proceeds to the callback. - Throw-path latch release: `db.all()` rejection still releases the latch in `finally`. - Latch precedence: directly-set `isClearing` suppresses even when shutdown gates would otherwise allow dispatch. - Uninitialized clear: short-circuit returns false without setting the latch. ## Validation - `npm run typecheck` — clean. - `npm run lint` — no new warnings or errors on the modified files. - `npx vitest run tests/unit/profile/` — 2287 passed (143 files). - `npx vitest run tests/unit/core/Sphere.clear.test.ts` — 15 passed.
…ispatch, UXF/TXF parse, Helia blockstore Item #3 of issue #364: instrument the remaining 47s of §D.4 wall-time in the SKIP_RULE4 soak. The existing perf-counters cover ~62 % of §D.4; this commit adds opt-in counters for the dominant uninstrumented paths (snapshot dispatcher, UXF/TXF parse, Helia blockstore writes/pins, OrbitDB read/write surface) so the next soak attributes >= 85 %. All counters are gated by SPHERE_PERF=1 via the existing core/perf-counters.ts helpers (incr / observeMs). Zero overhead when off — verified by smoke test. New counters added (29 total): profile/profile-snapshot-dispatcher.ts (runProfileSnapshotJoin): - profile.snapshotJoin.calls - profile.snapshotJoin.totalMs - profile.snapshotJoin.decodeMs - profile.snapshotJoin.decodedEntries - profile.snapshotJoin.perWriterDispatchCalls - profile.snapshotJoin.perWriterDispatchMs - profile.snapshotJoin.bundleIndexJoinCalls - profile.snapshotJoin.bundleIndexJoinMs profile/profile-snapshot-merge.ts (runJoinSnapshot per-entry): - profile.snapshotJoin.perKeyApplyCalls - profile.snapshotJoin.perKeyApplyMs serialization/txf-serializer.ts: - serialization.txfSerializer.txfToToken.calls / .totalMs - serialization.txfSerializer.parseStorageData.calls / .totalMs uxf/UxfPackage.ts: - uxf.ingest.calls / .totalMs - uxf.ingestAll.calls / .tokens / .totalMs / .perTokenDeconstructMs - uxf.assemble.calls / .totalMs profile/helia-blockstore-shim.ts (get/put/delete): - helia.blockstore.get.calls / .cacheHit / .cacheHitMs / .inflightHit / .inflightHitMs / .miss / .missMs / .bytes - helia.blockstore.put.calls / .totalMs - helia.blockstore.delete.calls / .totalMs profile/helia-blockstore-pin-shim.ts (helia.pins.add + putMany): - helia.pins.add.calls / .cached / .successMs / .alreadyPinned / .alreadyPinnedMs / .failed / .failedMs - helia.blockstore.putMany.calls / .items / .totalMs profile/orbitdb-adapter.ts (non-putEntry surface): - orbitdb.put.bundle / .other / .error - orbitdb.get.hitMs / .missMs / .errorMs / .error - orbitdb.del.totalMs / .error - orbitdb.getEntry.bundleHitMs / .hitMs / .bundleMissMs / .missMs / .errorMs / .error - orbitdb.all.calls / .dbAllMs / .entries / .totalMs / .error Smoke test: confirmed counters fire under SPHERE_PERF=1, snapshot() returns 0 keys when SPHERE_PERF is unset.
Replace the prior breadcrumb-based design (closed as fix/issue-364-item4-rule4-design-gate) with per-bundle provenance annotated at the snapshot dispatcher's writeRemote site. The gate's decision is now keyed on the loaded bundle set itself, not on provider-global mutable state — eliminating the §D.1 retry-storm staleness where periodic-poll applySnapshots re-armed the breadcrumb between disarm sites and the next load() incorrectly skipped Rule-4 enrichment. ## Design A new optional `sourcedFromSnapshotPointerCid` field on UxfBundleRef carries the IPFS CID of the lean-snapshot blob that placed the bundle ref into local OrbitDB. It is set ONLY when the snapshot dispatcher's `writeRemote` callback inside `BundleIndex.joinSnapshot` is the writer that landed the ref. Locally-published refs (`BundleIndex.addBundle`) and replication-arrived refs (OrbitDB pubsub) leave the field unset, as do legacy refs persisted pre-#367. At Rule-4 gate time in `_loadImpl`: skipRule4Snapshot = loadedBundles.length >= 2 && loadedBundles[0].sourcedFromSnapshotPointerCid !== null && loadedBundles.every(b => b.sourcedFromSnapshotPointerCid === loadedBundles[0].sourcedFromSnapshotPointerCid) Any non-snapshot bundle in the active set, or a mix of source snapshots, forces Rule-4 to run. The gate is robust against periodic- poll firing arbitrarily often, because the signal lives in the loaded data, not in any global provider state. Why this also fixes the §D.1 storm (the §D.1 V6-RECOVER probe loads() ran during periodic-polls firing applySnapshot on bob's wallet, which re-armed the original breadcrumb between disarm sites — the next load() then incorrectly skipped Rule-4 → finalize-side enrichment loss → 30s cooldown × 3 retries × N stranded tokens, §D.1 wall ballooned 98s → 1275s): - Periodic-poll's applySnapshot may land 0 new bundles (the wallet already has the bundles the snapshot encodes). The pre-existing local bundles carry no `sourcedFromSnapshotPointerCid` — they were written by `addBundle`. The gate sees mixed/missing provenance and KEEPS Rule-4 on. Correct. - In §D.4 fresh recovery: the wallet starts with empty OrbitDB, applySnapshot lands N bundles via writeRemote — every landed ref carries the same source CID. The gate fires correctly and skips Rule-4 — the soak speedup that motivated the original Item #4 is preserved. ## Plumbing - `profile/types.ts` — new optional field on UxfBundleRef. - `profile/profile-token-storage/bundle-index.ts` — `BundleIndex` gains `currentSnapshotApplyCid` + a `setCurrentSnapshotApplyCid` setter. `joinSnapshot`'s `writeRemote` decodes the envelope, decrypts, parses the ref, re-stamps with the source CID, re- encrypts, re-envelopes. Any failure in the annotation pass falls back to the verbatim write — the JOIN still lands and the gate degrades safely (Rule-4 runs). - `profile/profile-snapshot-dispatcher.ts` — `ProfileSnapshotJoinDeps` gains optional `sourcePointerCid: string`. Before invoking the BundleIndex JOIN, the dispatcher duck-types `setCurrentSnapshotApplyCid` on the bundle-index handle and arms it; clears in `finally` so a stale CID can never leak. - `profile/factory.ts` — `runProfileSnapshotApply` and `dispatchParsedSnapshot` accept and forward `sourcePointerCid`. Both apply paths (the periodic-poll `setApplySnapshotCallback` and the pointer-wired `setSnapshotApplier`) pass through the pointer CID they decoded at fetch time. - `profile/pointer-wiring.ts` — `applySnapshot` callback signature widens to `(snapshot, sourcePointerCid?)`. The `fetchAndJoin` callback already decodes the remote CID for the dag/get fetch; we now also pass it into the applier. - `profile/profile-storage-provider.ts` — `snapshotApplier` and `setSnapshotApplier` widen to accept the optional source CID. - `profile/profile-token-storage-provider.ts` — `loadedBundles` entries now carry `sourcedFromSnapshotPointerCid` (read from the bundle ref returned by `listActiveBundles`). The Rule-4 gate computes `skipRule4Snapshot` per the design above and ORs with the existing `SPHERE_SKIP_RULE4` env-gated path. New perf counter `profile.load.rule4SkippedSnapshot` fires when the gate skips. ## Tests `tests/unit/profile/load-rule4-snapshot-gate.test.ts` (new) — 5 cases pinning the gate's decision matrix at the `load()` layer: - all bundles share one source CID → skip; - all bundles have no provenance (locally-published) → no skip; - mixed snapshot-sourced + local → no skip; - bundles span two distinct source CIDs → no skip; - single-bundle snapshot-sourced load — the `>= 2` structural guard short-circuits the pairwise loop independently of the gate (gate is moot). The existing `tests/unit/profile/load-rule4-wiring.test.ts` still passes (4 cases) — its scenarios use bundles without the new field, so the gate stays off and Rule-4 fires exactly as before. Validation: - npm run typecheck — clean - npm run lint — clean (full repo, 0 errors / 0 warnings) - npm run test:run — 8805/8820 unit tests pass; 13 skipped; 2 pre- existing flakes in tests/integration/nametag-normalization.test.ts (parallel-execution TEST_DIR race documented in project_core_sphere_test_flake.md — passes 7/7 in isolation, unrelated to this change).
…liminate 108×/recovery over-invocation Issue #363 §D.4 (full mnemonic recovery, 123 s wall) observed 108 `pointerLayer.recoverLatest()` calls — 22 % of §D.4 wall — driven by `LifecycleManager.awaitAggregatorPointerReadBack` polling every 500 ms to wait for the aggregator to catch up to a just-published snapshot CID. Per the L4 model the design rate is one recoverLatest per recovery window; the read-back loop was paying the full aggregator walkback cost on every 500 ms tick. Approach chosen: (a) cache inside `ProfilePointerLayer`. There are six production call sites (lifecycle-manager read-back loops, cold- start recovery, periodic poll, WALKBACK_FLOOR reconcile, pointer-win broadcast handler), so a per-caller short-circuit would be error- prone. The cache sits INSIDE the existing `time()` wrapper so cache- hit and cache-miss latencies still count toward the aggregate `pointerLayer.recoverLatest` timer — the call count semantic is preserved while totalMs/avgMs drop dramatically. Mechanism: - `#cachedHead: { result, expiresAt } | null` with a 10 s default TTL (configurable via the new `recoverLatestCacheTtlMs` init option). Set `0` to disable. - `#inFlightRecover: Promise<…> | null` coalesces concurrent calls onto a single aggregator round-trip; the deduped waiters do NOT share the upstream call's abort signal (their own aborts are honored on entry). - Cache cleared on `shutdown()` so a re-init reads a fresh aggregator view; the in-flight slot is cleared by the operation's own finally arm during the existing `#inFlight` drain. Counters added (visible when `SPHERE_PERF=1`): - `pointerLayer.recoverLatest.cacheHit` — within-TTL hit - `pointerLayer.recoverLatest.cacheMiss` — TTL > 0, no cached - `pointerLayer.recoverLatest.cacheStale` — TTL expired - `pointerLayer.recoverLatest.inFlightDedup` — concurrent attach Projected impact on §D.4 read-back loop (60 s deadline, 500 ms poll, 250 ms per call): Before: 80 aggregator round-trips After (TTL=10 s): 6 round-trips + 111 sub-ms cache hits Reduction: ~13× The 10 s TTL is well inside the typical 30 s shutdown-durability deadline and well outside the 30–90 s periodic-poll cadence, so: - new pointer versions are still observed within 10 s of the aggregator surfacing them (acceptable for the read-back loop, whose existing budget is tens of seconds); - the periodic poll's recoverLatest always misses the cache (TTL expired between polls), so no behavioral change at that layer. Tests (`tests/integration/pointer/recover-latest-cache.test.ts`): - cold call → cacheMiss, warm call within TTL → cacheHit + no aggregator round-trip; - 5 concurrent calls → 1 round-trip + 4 inFlightDedup; - TTL expiry → cacheStale + cacheMiss + fresh round-trip; - TTL=0 disables the cache (pre-#364 behavior); - shutdown clears the cache and refuses further calls; - pre-aborted signal throws AbortError before reading the cache. All 290 existing pointer + lifecycle-manager tests stay green.
… recoverLatest Re-incarnates the closed Item #1 (commit e3959e3, cherry-picked into the preceding commit) with the missing publish-time invalidation hook that caused PR #365 to drop it. The §D.4 wall-clock degradation observed in PR #365's A/B (243 s → 467 s when the cache was enabled) traced directly to the read-back loop at `LifecycleManager.awaitAggregatorPointerReadBack` spending the full 10 s TTL serving a pre-publish CID after the local publish committed a new version. ## Mechanism Inside `ProfilePointerLayer.#publishInner`, after `reconcileAndPublish` returns without throwing (the publish committed a new pointer version that downstream readers must observe), the cache is cleared: if (this.#cachedRecover !== null) { incr('pointerLayer.recoverLatest.cacheInvalidatedOnPublish'); this.#cachedRecover = null; } Properties: - **Every successful publish path invalidates** — including the fast path (`result.attemptsUsed === 0`) that skips Step B (#263). The hook runs after the reconcile returns and is independent of which branch the reconcile took. - **Conditional on a populated cache** — the counter only fires when there's actually something to clear, so the signal is meaningful (a publish on a cold cache is a no-op). - **In-flight slot is intentionally not touched** — a recoverLatest already awaiting `#inFlightRecover` will still receive that round- trip's result (the in-flight slot is owned by the operation already in motion). A NEW recoverLatest after the publish starts fresh. - **No abort propagation** — the cache invalidation is local provider state; it does not interact with the in-flight operation's abort signal. ## Tests `tests/integration/pointer/recover-latest-cache.test.ts` gains three new cases at the bottom of the existing describe block: 1. **publish() invalidates the cache: subsequent recoverLatest is a fresh round-trip.** Seeds v=1, warms the cache, asserts cache-hit semantics, publishes v=2, asserts the `cacheInvalidatedOnPublish` counter fired, then asserts the next recoverLatest is a fresh aggregator round-trip surfacing v=2. 2. **publish() with no cached value is a no-op for the invalidation counter.** A publish on a cold cache must not bump the counter. 3. **publish() invalidation does not abort an in-flight recoverLatest started before publish.** Pins the in-flight semantics: a concurrent recoverLatest+publish pair resolves cleanly, and a fresh recoverLatest after both settle observes v=2. All 9 cache tests pass (6 original + 3 new). 587 + 9 skipped pointer tests across `tests/unit/profile/pointer/` and `tests/integration/pointer/` pass. ## Counters New: `pointerLayer.recoverLatest.cacheInvalidatedOnPublish` (bumped per successful publish that found a populated cache). Sits alongside the existing cacheHit / cacheMiss / cacheStale / inFlightDedup counters from the Item #1 cherry-pick. ## Refs - Closes #366 - Re-incarnates `fix/issue-364-item1-pointer-recover-cache` (commit e3959e3) cherry-picked in the preceding commit. - Original A/B data: `/tmp/soak-metrics/issue-363/REPORT.md`
…ng Sphere.clear Closes the multi-wallet gap left by Item #2 (PR #365). The per-instance `isClearing` latch correctly suppresses `applySnapshotIfWired` for THE provider whose `clear()` is in flight, but a sibling provider's periodic pointer-poll keeps firing — its own `isClearing` is false. The §D.3 soak baseline shows 23 `profile.applySnapshot.fired` events across the clear-all-wallets phase; PR #365 reduces this only for the actively-cleared wallet. This commit adds a process-wide reference-counted gate that `Sphere.clear()` brackets around its destructive body. While the bracket is held, EVERY provider in this process suppresses `applySnapshot` and bumps the new counter `profile.applySnapshot.suppressedDuringGlobalClear`. ## Design — separate leaf module The gate state lives in `profile/global-clear-gate.ts` (no exported class — three top-level functions over a module-private `depth` counter). Both `core/Sphere.ts` and `profile/profile-token-storage-provider.ts` import this leaf. Putting the state as a class-static on `ProfileTokenStorageProvider` would force `core/Sphere.ts` to import the provider class directly; the leaf module keeps the dependency direction clean. ## Semantics - **Reference counted.** Nesting composes — an orchestrator may bracket a multi-wallet batch in a higher-level pair while each `Sphere.clear()` body also brackets itself. - **No-op-safe at depth 0.** A stray double-end is harmless; `endGlobalClear()` only decrements when depth > 0. Prevents silent gate-stuck-closed failures from a buggy orchestrator. - **Process-scoped.** One counter per JS realm. Cross-process coordination is out of scope; filesystem / IndexedDB locks already serialize cross-process wallet wipes. ## Files - `profile/global-clear-gate.ts` (new) — `beginGlobalClear()`, `endGlobalClear()`, `isGlobalClearActive()`, plus `__resetGlobalClearForTest()` for test setup/teardown. - `profile/profile-token-storage-provider.ts` — `_applySnapshotIfWiredImpl` now checks `isGlobalClearActive()` AFTER the per-instance `isClearing` latch (preserving Item #2's counter attribution). When the global gate fires, it bumps `profile.applySnapshot.suppressedDuringGlobalClear` and returns null without invoking the wired callback. - `core/Sphere.ts` — `Sphere.clear()` wraps its destructive sequence (destroy instance + clear L1 vesting + IDB settle + token storage clear + fallback clear + KV storage clear) in `beginGlobalClear()` / `try` / `finally` / `endGlobalClear()`. The bracket is released even on a destructive failure inside the body so a partial clear never leaves the gate stuck closed. ## Tests `tests/unit/profile/global-clear-gate-368.test.ts` (new) — 7 cases: 1. begin / end track depth; `isGlobalClearActive()` reflects state. 2. Two begins require two ends to release (nesting composes). 3. `endGlobalClear()` no-op-safe at depth 0; subsequent begin still moves the gate to 1 cleanly. 4. Held gate suppresses `applySnapshotIfWired` across an UNRELATED provider whose `isClearing` is false — the multi-wallet scenario. 5. Counter `profile.applySnapshot.suppressedDuringGlobalClear` fires exactly once per gated call (3-of-3 in a tight loop). 6. Per-instance `isClearing` takes precedence — counters stay attributable when both gates would fire. 7. After the gate releases, `applySnapshotIfWired` proceeds normally. The Item #2 sibling tests (`profile-token-storage-clear-suppress-apply-snapshot-364.test.ts`) remain green (6/6) — the global gate is checked AFTER the per-instance latch, so existing semantics are preserved. ## Validation - `npm run typecheck` — clean - Targeted lint clean on the 4 changed files - `npx vitest run tests/unit/profile/ tests/unit/core/Sphere.clear.test.ts` — 2309 / 0 failed across 145 test files ## Refs - Closes #368 - Builds on PR #365 (`integration/issue-364-items-2-3-6`) — same base. - Sibling to PRs #373 (Rule-4 gate, issue #367) and #374 (recoverLatest cache, issue #366) — also close gaps from PR #365's dropped items.
…ULT_PIN_CONCURRENCY Two of the three improvements from #369. The third (expanding `DEFAULT_IPFS_GATEWAYS` from a single primary to operator-provided unicity-ipfs2/3/4/5 fallbacks) is intentionally out of scope here — that's a deployment / infrastructure decision the SDK can't make unilaterally; operators wire additional gateways via the `SPHERE_IPFS_GATEWAY` env override or the `ipfsGateways` config field the loop already iterates. ## 1. Retry-with-backoff for transient pin failures Adds `withPinRetry` + `isTransientPinError` helpers around the existing gateway-iteration loop in `pinToIpfs` and `pinSingleBlock`. Up to 4 attempts total (1 initial + 3 retries) with 100 / 500 / 2000 ms backoff between them — 2.6 s of accumulated backoff before exhaustion, well inside any operation's deadline budget. The classifier distinguishes transient (retry) from permanent (short-circuit): - Transient: `fetch()` throws (network blip, ECONNRESET, ETIMEDOUT, EAI_AGAIN, AbortError, TimeoutError), HTTP 5xx (gateway-side overload), HTTP 429 (explicit rate-limit signal). - Permanent: HTTP 4xx other than 429 (bad request, unauthorized, not found, payload too large) — these are deterministic client-side failures and never clear on retry. - Lenient default: unrecognised error shapes are treated as transient. The bounded backoff schedule caps the cost. The HTTP-status match scans the message for `HTTP NNN` anywhere (not just the prefix) so a wrapped failure (e.g. `ProfileError("IPFS pin failed on all gateways: HTTP 400 ...")`) still classifies on the inner status rather than falling through to the lenient default. This change makes #369 a no-op when every gateway returns a real 4xx — the budget is preserved for actual transient situations. New perf counters: - `ipfs.pin.retry.attempt` — bumped before each backoff sleep. - `ipfs.pin.retry.exhausted` — bumped when all retries fail transient. - `ipfs.pin.retry.permanent` — bumped on a permanent short-circuit. ## 2. Lower DEFAULT_PIN_CONCURRENCY 10 → 5 The Unicity kubo container caps at `MAX_PINS_PER_SECOND=100`. With 10 concurrent slots × ~10 pin/s/slot the SDK peaked at ~100 pin/s and routinely brushed the limit on bursts, producing intermittent `IPFS dag/put failed on all gateways for <CID>: fetch failed` surface errors that the soak couldn't reproduce on demand because the gateway worked ~99% of the time when tested directly. 5 concurrent slots stay comfortably under the cap (~50 pin/s peak) while still parallelizing meaningfully — the 250-block migration CAR pays ~5 s instead of ~2.5 s, but with substantially fewer rate-limit- induced retry storms. Combined with #1, a single ~1% transient block failure across a 250-block CAR is now mathematically a non-event (0.99²⁵⁰ ≈ 8% no-failure baseline → with 3 retries, the per-block success rate becomes 1 - 0.01⁴ ≈ 99.99999999%, and the CAR-level success rate is essentially indistinguishable from 100%). ## Tests `tests/unit/profile/ipfs-client-pin-retry-369.test.ts` (new) — 14 cases covering: - `isTransientPinError` classifier across the failure-mode matrix the pin path actually produces (5xx / 429 / 4xx / fetch errors / AbortError / unrecognised shapes). - `withPinRetry` retry / backoff / counter semantics: first-call success skips retries, transient failures retry until success, exhausted retries throw, permanent errors short-circuit, default backoff schedule is [100, 500, 2000] ms. - End-to-end `pinToIpfs` proves the wrap is in the right place: succeeds on third attempt after two transient failures (4 fetch calls = 2 failed + 1 success + 1 sidecar submit), exhausts at 4 attempts × 1 gateway when all-transient, and HTTP 400 short- circuits to 1 attempt only. `tests/unit/profile/ipfs-client-parallel-pin.test.ts` — pre-existing abort-propagation tests that intentionally trigger a single block failure are switched from HTTP 500 (now classified transient and retried) to HTTP 400 (still classified permanent and short-circuits), preserving the original test intent ("a single block failure aborts the whole CAR pin") under the new retry-with-backoff wrap. The 6-character source diff is byte-equivalent for `parallelPin` behaviour; the test docstring is otherwise unchanged. `tests/unit/profile/lifecycle-manager-pointer-poll.test.ts` — the "returns a NEW snapshot CID → applier is invoked, re-arms" case runs under `vi.useFakeTimers()`, then triggers a flush whose CAR-pin retry hangs on fake-timer setTimeouts. Switched to `vi.useRealTimers()` for the teardown shutdown so the retries drain promptly — the test's assertions (applier was invoked with the new snapshot CID, poll re-armed) all complete BEFORE the shutdown call; the real-timer switch only affects resource-cleanup discipline. The test's elapsed wall-clock grows from 15 ms to ~5.5 s due to the real retry backoffs; this is acceptable for a single integration case. ## Validation - `npm run typecheck` — clean - Targeted lint clean on the 4 changed files - `npx vitest run` — 8816 passed | 13 skipped | 0 failed across 539 test files ## Acceptance criteria (per issue) - [x] Single transient gateway failure no longer aborts a CAR pin — wrapped by `withPinRetry`, 3 retries with backoff. - [ ] Soak passing rate ≥ 95% across 10 consecutive runs — soak A/B is a separate run after this lands; happy to attach numbers in a follow-up comment. - [ ] §D.1 wall-clock comparable to baseline (~98 s) without rate- limit-induced retry storms — measured in same follow-up. ## Refs - Closes #369 - Discovered during #364 soak validation; multiple soak runs hit `fetch failed` against `unicity-ipfs1.dyndns.org` while the gateway tested as reachable (HTTP 200) within seconds before/after. - Kubo container is healthy (247 peers, 3 MB/s bandwidth). - Code sites: `profile/ipfs-client.ts:448` (`pinToIpfs`), `:691` (`pinCarBlocksToIpfs`).
…ort with capability probe and legacy fallback Replaces the SDK's per-block IPFS round-trip pattern (`/api/v0/dag/put` per block, `DEFAULT_PIN_CONCURRENCY=10` parallel HTTP connections) with single-round-trip CAR-level push and pull via Kubo's `/api/v0/dag/import` (push) and `/api/v0/dag/export` (pull) endpoints. Eliminates the throttling-under-burst behaviour that caused several spurious soak failures during #364 validation (kubo container's `MAX_PINS_PER_SECOND=100` limiter was being tipped over when multiple wallets flushed simultaneously, surfacing as `IPFS dag/put failed on all gateways: fetch failed`). SDK-side changes only — operator-side gateway config (haproxy ACL + rate-limit revisit on `unicity-ipfs1.dyndns.org`) is "Part 1" of #370 and lives outside this repo. The capability probe makes the SDK degrade gracefully on legacy gateways without `/dag/import` and `/dag/export` exposed; once the operator flips the gateway config, the SDK auto-detects and uses the fast path. Public API unchanged. `pinCarBlocksToIpfs` and `fetchCarFromIpfs` become "probe + dispatch" selectors: * `pinCarBlocksToIpfs` first probes the gateway. If `/dag/import` is exposed, parses the CAR locally for the #236 local-Helia writes (every block written before HTTP push), then issues a SINGLE multipart POST of the CAR to `/dag/import?pin=true`. Parses the NDJSON response permissively (accepts both `{"Root":{"Cid":{"/":"..."}}}` and `{"Root":{"Cid":"..."}}` shapes) and verifies our expected root is present with no `PinErrorMsg`. On any fast-path failure, falls through to the legacy per-block `/dag/put` loop with `helia=undefined` so the legacy path does NOT duplicate the local-Helia writes already performed. * `fetchCarFromIpfs` short-circuits to the legacy BFS when local Helia already has the root (the BFS is local-Helia-aware via `fetchFromIpfs` and walks zero HTTP for fully-cached CARs). Then probes. If `/dag/export` is exposed, issues a SINGLE POST and parses the returned CAR with end-to-end per-block CID verification (`verifyCidMatchesBytes` on every block — preserves the gateway- tampering boundary). Reassembles a CARv1 in stream order matching the existing BFS output shape. Falls through to legacy BFS on any failure (probe miss, mid-call HTTP error, CID-binding violation, missing root in the exported CAR). Capability probe is a tiny no-body POST to `/api/v0/dag/{import,export}` with a 2-second timeout. Response classification: - 4xx (except 404, 405) and 2xx → exposed (Kubo returns 400 for empty body on a healthy import endpoint). - 404, 405, 5xx → not exposed (404 = route absent, 405 = POST blocked at proxy, 5xx = upstream unhealthy → safer to skip). - Network error / timeout → not exposed. Cached per-process per normalized gateway URL — `_resetGatewayCapabilityCache()` exported for tests. Sidecar submit: one fire-and-forget POST per CAR root on the fast path (issue-#370 explicit "called once per CAR root" guidance); unchanged per-block on the legacy path. Tests: * `tests/unit/profile/ipfs-client-dag-import.test.ts` (14 tests) — probe semantics (200/404/405/500/network-error), probe cache, permissive NDJSON parsing (modern + bare-string Cid + mixed Stats/Root lines), multi-root defense, fallback on missing-root / PinErrorMsg / 500, local-Helia write deduplication. * `tests/unit/profile/ipfs-client-dag-export.test.ts` (7 tests) — happy path single-shot, probe-miss fallback, mid-call 500 fallback, missing-root defense, CID-binding-violation defense, byte-equivalence with legacy BFS, probe cache. * All existing IPFS tests pass unchanged (54 tests across `ipfs-client-parallel-pin`, `fetchCarFromIpfs`, `ipfs-client-sidecar-submit`, `ipfs-client-cid-verify`, `ipfs-client-helia-blockstore-236`). * Full `tests/unit/profile/` suite green (2302 / 2302 tests). Perf-counter wiring (`ipfs.dagImport.totalMs`, `ipfs.dagExport.totalMs`) deferred — `core/perf-counters.ts` is on `chore/perf-instrumentation` (chained via PR #365) and has not landed on `main` yet. Will be added as a follow-up once that lands. Blocks the throttling-mitigation half of #369 (per #370 "Ordering / blocks" section): once #370 lands, the per-block path is the legacy fallback only and #369's per-block retries become much less load-bearing. Refs: #370, #364, #369, #236, #255, #200
…r live A/B
Adds a sibling to the existing _resetGatewayCapabilityCache test/soak
helper. Lets a soak probe pre-populate the capability cache for a
specific gateway URL, so an A/B comparison can run both the legacy
and fast paths against the SAME live gateway (forced-legacy on side A
by injecting {dagImport: false, dagExport: false}; allowed to probe
naturally on side B).
Used by .tmp/soak-370/ab-compare.mts (not tracked) to produce the
live-gateway A/B that validates Part 1 of #370 end-to-end.
Live A/B result (Side A: forced legacy, Side B: fast path), 250-block
CAR / 51 KB total, 3 iter each, against unicity-ipfs1.dyndns.org with
Part 1 (the nginx ACL change) deployed:
pin avg fetch avg speedup
A: 1391.3 ms 3763.7 ms -
B: 17.3 ms 17.3 ms pin 80×, fetch 217×
Exceeds the #370 acceptance criterion (5-10×) by an order of
magnitude. A second back-to-back run of side A hit
`SocketError: other side closed` mid-/dag/put burst — i.e. kubo's
per-block API surface gets overwhelmed by the 10-connection
parallelism per call, which is the EXACT throttling/capacity failure
mode the fast path eliminates by design (single connection, single
round-trip, single body).
…t-circuit drain P4 from #275's optimization list, tracked as #378 after #275 closed on P1+P2 landing. Once `finalizeStrandedReceivedToken` classifies a stranded receive as permanent (HD-index recovery exhausted or permanent structural failure), the tokenId is recorded in a persistent ledger. Subsequent `drainPendingFinalizations` invocations skip these tokens in <100 ms instead of repeating the 60-s drain timeout per token. The #275 forensics measured this overhead at 30-60 s per `sphere balance` invocation while a stranded token exists. ## Files - `constants.ts` — new `STORAGE_KEYS_ADDRESS.V6_RECOVER_PERMANENT` key (`v6_recover_permanent`). - `modules/payments/PaymentsModule.ts`: - **Field** — `v6RecoverPermanent: Map<string, { reason: string; ts: number }>` tracks the verdict. - **Persistence helpers** — `saveV6RecoverPermanent()` / `restoreV6RecoverPermanent()` mirror `saveProofPollingJobs` / `restoreProofPollingJobs`: empty map → `storage.remove()` so stale lists never survive; malformed payloads log + start empty; individual malformed entries are silently skipped. - **Mark site** — V6-RECOVER permanent path (`~line 9778`): after the existing status='invalid' write, stamp the ledger with the canonical `classLabel` reason + `Date.now()`. Defense-in-depth above the status flip: a `load()` that re-derives the in-memory token map from disk could (depending on storage-layer status- merge semantics) restore the original 'submitted' status; the persistent ledger key is independent of token bytes and survives that round-trip deterministically. - **Drain skip** — `hasUnconfirmedOrInflight` (`~line 7877`): a token whose tokenId is in the ledger is excluded from the drain predicate even if its status reverted to 'submitted' / 'pending'. - **Stranded-scan skip** — `recoverStrandedReceivedTokens` (`~line 9358`): the V6-RECOVER cold-start scan checks the ledger BEFORE re-deriving sender predicates / requestId / commitments, so a previously-permanent token doesn't re-pay the multi-second probe + finalize cycle on every CLI cold start. - **Forced-retry clear** — `receive({ finalize: true })` clears the ledger before drain so a stranded token gets one more shot at finalization (in case HD-index recovery has since widened). If the underlying condition genuinely hasn't cleared the recovery path re-derives the verdict and re-stamps the ledger. - **Restore on load** — `restoreV6RecoverPermanent` is invoked from `load()` AFTER `loadFromStorageData` populates `this.tokens` and BEFORE `recoverStrandedReceivedTokens` runs, so the scan sees the marker and skips immediately. ## Tests `tests/unit/modules/PaymentsModule.v6-recover-permanent-378.test.ts` (new) — 5 cases pinning the ledger's contract at the unit level: 1. `saveV6RecoverPermanent` + `restoreV6RecoverPermanent` round-trip preserves the marker across a simulated process boundary. 2. Empty-map save clears the underlying storage key (no stale list survives). 3. Malformed entries inside an otherwise-valid payload are silently skipped; well-formed entries restore correctly (single-entry resilience). 4. Missing storage key on restore is a no-op. 5. Non-array payload on restore logs + leaves pre-existing in-memory state untouched. The end-to-end V6-RECOVER finalize path itself is exercised in the existing integration tests under `tests/integration/payments/ v6-recover-real-sdk-recovery.test.ts`; this commit's new tests pin only the ledger semantics around it. ## Validation - `npm run typecheck` — clean - Targeted lint on the 3 changed files — 0 errors (5 pre-existing warnings in `PaymentsModule.ts` unrelated to this change). - `npx vitest run tests/unit/modules/` — 1779 / 0 failed across 100 test files. - `npx vitest run` — 8806 passed | 13 skipped | 1 pre-existing flake (`tests/integration/wallet-clear.test.ts:217:26` — known parallel-execution `Wallet already exists` flake; passes 14/14 in isolation; documented in memory `project_core_sphere_test_flake.md`). ## Acceptance per #378 - [x] Once a token has hit V6-RECOVER permanent, subsequent `drainPendingFinalizations` invocations skip it in <100 ms — the predicate is an O(1) `Map.has` check. - [x] Marker survives process restart — `restoreV6RecoverPermanent` rebuilds the in-memory map from the persisted KV blob on `load()`. - [x] Marker is cleared by `Sphere.clear()` — the underlying KV key goes with the full wallet wipe (handled by the existing `storage.clear()` path, no extra code needed). - [x] Marker does NOT survive `payments receive --finalize` — forced-retry path clears the ledger before drain. - [x] Unit test exercises a stranded token persisting across two distinct `PaymentsModule` instances against the same storage. ## Refs - Closes #378 - #275 — primary issue, P1 + P2 landed via PR #277. This is P4. - Forensics: `/tmp/soak-274-c-debug/OPTIMIZATION-FINDINGS.md`.
…restore The log claimed to be clearing the persisted ledger on malformed payloads, but the function only logs and returns — the existing unit test (restoreV6RecoverPermanent on non-array payload clears nothing and logs) asserts this preserve-in-memory behavior intentionally. Match the log text to the actual behavior so future readers (and ourselves in incident review) aren't misled. Surfaced by adversarial pre-merge review of the perf-stack-with-370-379 integration.
… GroupChat #334 `CommunicationsModule.parseMessagesPayload` used to fatal-throw `ProfileError(CID_REF_UNREADABLE)` when the `messages` KV held a CID-ref envelope but the wallet was opened without a `cidRefStore` (legacy factory path). The throw propagated through `Sphere.load`'s `Promise.allSettled` as "Module load failed" — taking down every other module's load with it on the live testnet deploy 2026-06-01. Mirrors the GroupChat fallback added in PR #334 / commit 7421386: a `[CID_REF_DEGRADE]` warn + empty-state return. NIP-17 relay re-delivery rehydrates whatever the relay still retains. Also wraps the `cidRefStore.fetchJson()` call in try/catch so a gateway 404/timeout during fetch no longer bricks the load either — same recovery strategy. The previous fatal-throw was added because "silent fallback would mean silently losing all stored DMs for this address." That's still true, but a fatal throw also loses every other module's state and is strictly worse for the user. Warn + empty + relay-rehydrate retains the visibility without the load-bricking failure mode. Tests: - Converted `throws CID_REF_UNREADABLE …` → `degrades to empty messages …` - New: degrades on `cidRefStore.fetchJson()` rejection (gateway failure) - 74/74 CommunicationsModule tests pass
… console storm Live testnet deploy 2026-06-01 logs ~100 visible GET https://unicity-ipfs1.dyndns.org/sidecar/blob?cid=… 404 errors per wallet load. The SDK is already silent on 404 (`tryReadFromSidecar` returns null without logging), but Chrome auto-logs every non-2xx `fetch()` to the console. With ~100 blocks in a typical Profile snapshot the noise drowns real signal. Root cause is by design: the sidecar correctly GCs blobs once promoted to Kubo (`instant_pin_cache._reconcile_once` unlinks the disk blob after `block_stat` confirms in-kubo). So for any wallet load that walks an *old* Profile bundle every probe 404s — verified by curling the same CIDs and getting 200 from the same gateway's `/ipfs/` endpoint. Fix: per-gateway adaptive cold-cache. After SIDECAR_COLD_MISS_THRESHOLD (3) consecutive 404s on a gateway, both `tryReadFromSidecar` and `submitToSidecarBestEffort` short-circuit *before* issuing the fetch for SIDECAR_COLD_DURATION_MS (60 s). A successful hit (or 200 submit, which usually means a sibling device just published) resets the counter and re-arms the fast path immediately. Cross-device fast-path preserved: a single hit re-enables probing for the rest of the session. Steady-state cold gateways collapse from ~100 misses to ~3 misses + 1 re-probe per minute. 503 (cache_full back-pressure) and 413 (oversize) do NOT count as cold misses — the sidecar is alive in those cases, just refusing this particular submit. We must keep probing. `_resetSidecarColdState()` exported for test isolation, mirroring the existing `_resetGatewayCapabilityCache` pattern. Tests: - submit: short-circuits after N consecutive 404s - submit: 200 hit resets the counter - submit: 503 does NOT count as a cold miss - read: short-circuits after N consecutive 404s - read: hit resets the counter - re-arms after the cooldown window expires - 6/6 new tests pass
fix(communications)(page-freeze): degrade CID_REF_UNREADABLE — mirror GroupChat #334
perf(profile/ipfs): adaptive cold-cache for sidecar probes — kill 404 console storm
Audit #333 C1 was closed at the encrypt() boundary: when the OrbitDB write path was reachable but no encryption key had been derived, encrypt() threw and the write was rejected. That defense holds only during Phase A. After setIdentity attaches the encryption key, any subsequent write of `mnemonic` / `master_key` / `chain_code` / `derivation_path` / `base_path` / `derivation_mode` / `wallet_source` / `current_address_index` would have been encrypted and written to OrbitDB → replicated to IPFS via the snapshot CAR pin path. Even encrypted, distributing the seed lowers the threat model from "attacker must compromise the device" to "attacker must brute-force a password against an IPFS-pinned ciphertext" — strictly weaker than what users expect of seed material. The only legitimate cross-device transport for the seed is the user's own BIP-39 mnemonic backup. Fix --- - profile/types.ts: - New `IDENTITY_KEYS` set names the identity / seed-material legacy keys explicitly. - `CACHE_ONLY_KEYS` is widened to fold in `IDENTITY_KEYS`. After `translateKey()`, identity writes return `cacheOnly: true`; the `set()` flow short-circuits at the localCache step and never reaches `writeEnvelope()`. - profile/profile-storage-provider.ts: - Defense-in-depth: a post-translation assertion in `set()` fail-closes if any future refactor adds an identity-shaped Profile key (`identity.*`) without putting its legacy alias into `CACHE_ONLY_KEYS`. The error message points at the right file to fix. The old "encrypt() is the catch-all" comment block is replaced with one that documents the two-layer defense (cache- only routing + post-translation assertion). - profile/migration.ts: - Identity keys are diverted at read time into a separate `identityKeys` map and persisted at the LEGACY key name (so the cache-only routing kicks in). This keeps the legacy-import path working — Sphere.loadIdentityFromStorage reads `_storage.get('master_key')` against the Profile localCache and finds it under the same legacy name it always lived at — without routing any encrypted seed bytes through the OrbitDB OpLog. - `MigrationResult.keysMigrated` now counts identity keys too for parity with previous behaviour. - profile/index.ts: export the new `IDENTITY_KEYS` constant. Tests ----- - New test file `profile-storage-provider-identity-keys-cache-only.test.ts`: - Schema-level contract: every key in IDENTITY_KEYS is also in CACHE_ONLY_KEYS, and every `identity.*` mapping in PROFILE_KEY_MAPPING has its legacy alias in IDENTITY_KEYS. This catches future refactors at test time, not at runtime. - Parametrised assertion that `set('<identity-key>', ...)` writes to localCache and leaves OrbitDB empty. - Defense-in-depth assertion fires on a synthetic `identity.*` profile key with no CACHE_ONLY_KEYS entry. - Control: a non-identity write still flows through writeEnvelope. - `profile-storage-provider-c1-plaintext-seed.test.ts`: updated tests that asserted the OLD behaviour ("after setIdentity, mnemonic lands ENCRYPTED in OrbitDB"). The defense moved from "encrypt() is the boundary" to "cache-only is the boundary"; assertions now verify the seed never lands in OrbitDB at any point. - `profile-storage-provider.test.ts`, `integration.test.ts`, `migration.test.ts`, `migration-c2-flush-before-cleanup.test.ts`, `profile-storage-provider-issue-311.test.ts`: tests that used `'mnemonic'` as a generic test fixture for the OrbitDB write/read path swap to `'wallet_exists'` (non-identity, non-cache-only). Migration assertions verify the legacy key landed in Profile localCache (not under `identity.*`). Behaviour change consumers should know about -------------------------------------------- - Cross-device "wallet exists by virtue of identity.* being replicated" no longer works — Sphere.exists() on a fresh device returns false until the user imports the mnemonic, which is the correct UX for a non-replicating seed. The `has('wallet_exists')` fallback in ProfileStorageProvider still scans for legacy `identity.*` keys, so wallets created BEFORE this PR (which still have those entries in OrbitDB) keep working unchanged. - Identity material in legacy IndexedDB is migrated to the Profile localCache (IndexedDB, same device). The user-facing experience is unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
…o primary Symptom (2026-06-02): on a wallet that predates the IDENTITY_KEYS ⊂ CACHE_ONLY_KEYS fix, the seed material lives only in the legacy IndexedDB (wired as `fallbackStorage`). Every boot reads the primary (Profile localCache), finds nothing, falls back to legacy, succeeds, and emits one warning per identity key per boot: [Sphere] Identity read for "master_key" missing from primary storage; consulting fallbackStorage (legacy cached identity). The wallet works correctly but the warning noise is permanent — the fallback consult never goes away because nothing ever populates the primary. Fix --- In `Sphere.loadIdentityFromStorage`'s `readIdentityKey` helper, when the fallback read succeeds, also write the value back to the primary (`storage.set`). With the IDENTITY_KEYS ⊂ CACHE_ONLY_KEYS routing in the prior commit, the write lands in the Profile localCache only — it never reaches OrbitDB → never replicates to IPFS. So the backfill silences the per-boot warning for legacy wallets WITHOUT re-introducing the OrbitDB seed-leak the cache-only routing closes. The backfill is best-effort: if the primary `set` throws (e.g., quota / contention), the read has already succeeded — we must not regress the load just because the backfill couldn't run. Failure is logged at `debug`. Tests ----- New file `tests/unit/core/Sphere.identity-fallback-backfill.test.ts`: - backfill happens for every identity key the primary lacks; - backfill failure is swallowed (caller sees the original load-path error, not "quota exceeded"); - no backfill or fallback consult on the steady-state post-backfill boot (primary already holds the keys). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
…des ledger; soak gate V6-RECOVER permanent-mismatch verdicts were silently downgraded back to 'pending' on every load: `determineTokenStatus` (in txf-serializer) only encodes `pending`/`confirmed` from the TXF transactions array, so the application-level `'invalid'` write was lost on the next reload. `aggregateTokens` then surfaced the unspendable token as unconfirmed UCT, polluting balance across repeated `payments receive --finalize` calls. The persistent `v6RecoverPermanent` ledger added in #378 was correct on disk but never consulted at the balance / re-ingest surfaces. This commit makes the ledger authoritative without changing the TXF format: * New helper `applyV6RecoverPermanentInvalidStatus()` walks `this.tokens` and patches matching entries to `status='invalid'`. * Invoked at the end of every `loadFromStorageData` (initial load + every `sync()` round-trip) and at the end of `restoreV6RecoverPermanent` (handles cold-start ordering where the ledger hydrates AFTER the token map). * `aggregateTokens` adds a belt-and-braces ledger filter so balance NEVER includes a ledgered token, even if a future caller mutates status independently. * `addToken` honors the ledger on Nostr at-least-once replay — incoming tokens whose canonical id matches the verdict are persisted as `'invalid'`. We accept the write (return true) so the Nostr cursor advances; rejecting would cause perpetual replay. * Ledger lookups are canonical-id-first via the new `isV6RecoverPermanentToken(token, mapKey?)` helper — resolves via `extractTokenIdFromSdkData(sdkData)` with map-key fallback so UUID-keyed `addToken` entries still classify correctly. * `finalizeStrandedReceivedToken` keys the ledger by canonical genesis tokenId (was: map key, ambiguous post-replay) and awaits `saveV6RecoverPermanent()` so a process exit immediately after the verdict cannot lose the entry. Tests: * tests/unit/modules/PaymentsModule.v6-recover-invalid-status-387.test.ts — 16 unit tests covering helper mechanics, getBalance filter, restoreV6RecoverPermanent re-application, loadFromStorageData round-trip, addToken honoring ledger, and all 4 acceptance criteria from #387. * tests/integration/payments/v6-recover-invalid-status-387.test.ts — 4 e2e tests across the full public `module.load()` lifecycle with simulated CLI restarts. * Both files fail without the fix (verified via `git stash`) and pass with the fix. Soak hardening (`manual-test-full-recovery.sh`): * New `assert_no_unconfirmed_after_finalize` gate invoked at 5 post-`receive --finalize` sites (§A.1, §A.2, §C.2, §D.4 alice, §D.4 bob) — would have caught #387 on the first run. * `normalize_snapshot` strips `(+ N unconfirmed)`, `[X+Y tokens]`, and `(N tokens)` clauses so existing `assert_diff_empty` calls compare CONFIRMED balances only. Unconfirmed pollution is caught by the dedicated gate, not masked equally on both sides of a diff. * Self-tests T8 + T9 added; all 9 normalize self-tests pass. Soak run on testnet (779s, ALL GREEN): * All 14 ASSERT OK including 5 new #387 gates. * Zero address-mismatch / VerificationError / stranded receive diagnostics — the fix held end-to-end across `sphere clear` + full recovery, the exact scenario from the issue. Closes #387.
…9-quick-wins Wave 6-P2-9: quick wins — .env.example + soak-invoice-return + docs
…files, 70 cases Extends coverage over PaymentsModule surfaces that wave-6-P2-5 quarantined and wave-6-P2-7 left as follow-ups. All tests exercise the slim v2 rebuild via a fake in-memory ITokenEngine — no network, no SDK, no crypto. Added: - tests/unit/modules/payments/__fixtures__/v2-harness.ts (519 LoC) Shared harness: fake engine, recording token storage provider, captured transport handler, mint helper, buildIncomingTransfer helper. - PaymentsModule.cascade.v2.test.ts (9 cases) — split change back to self, cascade of change-then-send, multi-source greedy planCoinSpend, transfer-vs-split routing, sender-owned change output. - PaymentsModule.finalization.v2.test.ts (10 cases) — confirmed status on receive, verify() rejection, receive() + resolveUnconfirmed() emptiness, replay idempotence, memo propagation, malformed CBOR tolerance. - PaymentsModule.history.v2.test.ts (10 cases) — SENT/RECEIVED entry shape, sort order, defensive copy, addToHistory, storage roundtrip. - PaymentsModule.tombstone.v2.test.ts (11 cases) — removeToken + tombstone accumulation, mergeTombstones dedup, pruneTombstones no-op, storage roundtrip. - PaymentsModule.importExport.v2.test.ts (10 cases) — exportTokens, addToken/updateToken/onTokenChange, archived/forked maps, storage roundtrip, importTokens() stub flagged as BUG. - PaymentsModule.proofPolling.v2.test.ts (11 cases) — every install* no-op, atomic-return send (no polling loop), detectOrphan/ importInclusionProof/revalidate stubs, worker abort signal. - PaymentsModule.uxfAutoIngest.v2.test.ts (9 cases) — onTokenTransfer wiring, transport-authenticated senderPubkey, mixed-ownership bundle, empty CAR, CID payload deferral, back-to-back events. Suite delta: 7036 → 7106 (+70 passing). tsc --noEmit clean. BUG FLAG: importTokens() drops the caller's payload and returns three empty arrays (documented in importExport.v2.test.ts). If any consumer relies on it to hydrate a v2 blob into the wallet, that hydration path is silently broken. Flag for a follow-up wave.
…iles, 86 cases
Wave 6-P2-5 quarantined the entire pointer-layer legacy test suite when the
stsdk-v1 imports could no longer resolve in the v2 harness. This wave
restores the highest-value invariants against the current pointer surface,
mocking the aggregator client rather than hitting real network.
New files (tests/unit/pointer/):
aggregator-probe.v2.test.ts (21 cases)
- probeVersion H2 OR-predicate: both/one/neither side OK
- Rotation vs forgery: TRUST_BASE_STALE / UNTRUSTED_PROOF
- AbortSignal propagation (PointerProbeAborted)
- classifyVersion four-way: VALID / SEMANTICALLY_INVALID /
PROOF_TRANSIENT / CAR_TRANSIENT
- SDK shape drift → PROTOCOL_ERROR
- decodeVersionCid ok/transient/semantic
- isReachable: JsonRpcNetworkError, JsonRpcError treated as reachable
aggregator-submit.v2.test.ts (30 cases)
- classifySideResult per-side mapping: SUCCESS, EXISTS, REJECTED,
429, 5xx, 4xx, -32006, SyntaxError, generic Error
- combineOutcomes §7.3 rows 1-15 (state machine coverage)
- Row 4 vs 5 disambiguation via isIdempotentRetryHint
- Row 6/7 retry_side with committedSideKind='success'/'exists'
- retry_after cap at 600s
discover-algorithm.v2.test.ts (17 cases)
- Phase 1: validV=0 pristine wallet
- DISCOVERY_OVERFLOW when probes hit hard ceiling
- Input validation: currentLocalVersion, walkbackLimit,
initialEpochFloor — all raise PROTOCOL_ERROR
- Deadline diagnostic (#450): past-deadline message
- Pre-aborted signal → RETRY_EXHAUSTED
- computeProbeFingerprint: empty, deterministic, order-independent
signing.v2.test.ts (12 cases)
- Shape: 33-byte compressed pubkey, 66-char lowercase hex
- Determinism across builds; cross-wallet distinctness
- Signing round-trip: sign + verifyWithPublicKey ok / rejects wrong key
- T-A8 discipline: createFromSecret differs from raw constructor
- bytesToHex utility
publish-recover-roundtrip.v2.test.ts (6 cases)
- submitPointer → decodeVersionCid round-trip at v=1, 42, 7
- CID_MAX_BYTES=63 boundary; single-byte CID
- Cross-version isolation (v=2 unpublished → transient)
- Cross-wallet isolation (seed B fails on seed A's slot)
- Sequential publishes at v=1 + v=2 both decode independently
Constraints honored:
- No source edits (test-only wave)
- npx tsc --noEmit: 0 errors
- npx eslint tests/unit/pointer/*.v2.test.ts: 0 errors
- Full suite: 437 files / 7122 tests pass (was 431 / 7036)
…10a-payments-tests Wave 6-P2-10a: PaymentsModule v2 test coverage — 7 files, 70 cases
…teways
Wave 6-P2-7's coverage report flagged that the slim receive path REJECTS
`uxf-cid` payloads outright — returning false so the transport retains
the event indefinitely. That drops delivery for any sender using the
CID-by-reference envelope (per §3.3 of the UXF wire spec) since a fatter
consumer never arrives to drain the retention buffer.
Wire the recipient side:
1. If `deps.cidFetchGateways` is empty/null → keep the reject behavior
(return false, transport retains). Operators must configure
gateways to enable CID mode.
2. Otherwise, `fetchCarByCid` walks the gateway list in order and
verifies the CAR root CID matches `payload.bundleCid` (existing
pipeline module, already fully tested).
3. On successful fetch, parse the CAR with `CarReader.fromBytes`,
extract the single root block bytes, and feed them through the
same `extractInlineTokenBlobs` helper the inline `uxf-car` path
uses — so the downstream isOwnedBy → verify → persist tail is
bit-identical.
4. On any fetch failure → return false (retention, matches existing
transport contract).
5. On structural CAR/root-block failure after a successful fetch →
return null (retention) so a re-fetch can retry against a
healthier gateway.
Gateway URLs are never surfaced to logs — the top-level SphereError
message shape is bounded to `fetchCarByCid: ...` and we truncate
downstream error messages to 200 chars as defense-in-depth against a
gateway URL leaking through a nested cause.
Compilation clean (0 tsc errors). Test coverage lands in the next
commit.
Wave 6-P2-7 flagged two runtime paths in the slim rebuilds:
**Fix 1 audit — UnicityAggregatorProvider.waitForProof.**
The polling loop is correct: it terminates on success, sleeps
`pollInterval` between polls, and throws SphereError(TIMEOUT) on
deadline. No fix required. Adds `waitForProof.v2.test.ts` covering:
- immediate return when getProof resolves on the first poll
- proof:received event emission
- multi-poll cadence with onPoll attempt-counter increments
(fake timers driving the sleeps)
- SphereError(TIMEOUT) on deadline
- propagation of unexpected exceptions from getProof (no silent
swallow — operators need the diagnostic surface)
**Fix 2 — PaymentsModule uxf-cid receive path.**
Adds `PaymentsModule.uxfCid.v2.test.ts` covering the receive-path
behavior wired in the previous commit:
- happy path — mocked fetchCarByCid returns a real single-block
CARv1 (built with `@ipld/car`), root block bytes are the same
JSON envelope the inline uxf-car path consumes, receive tail
persists the token and emits `transfer:incoming` with memo +
senderNametag
- empty cidFetchGateways → returns false, no fetch attempted
- undefined cidFetchGateways (never wired) → returns false
- all-gateways-failed error → returns false (retention)
- CID-mismatch (surfaces as all-gateways-failed) → returns false
- truncated CAR (structural failure) → returns false
- CAR whose root block is not a JSON envelope → returns true (no
retention; peer sent junk, retry can't fix it)
- gateway URL redaction: log output stripped of https?://... tokens
Also tightens the URL-redaction in `fetchAndExtractCidBundle` (strip
URL substrings from the raw error message before logging) — the
"never expose gateway URLs" constraint is now enforced with an
assertion.
Updates the pre-existing `PaymentsModule.receive.v2.test.ts` uxf-cid
test description to reflect that the reject behavior now covers only
the "no cidFetchGateways configured" case.
Suite: 434 files, 7049 passed + 6 skipped, 0 failures.
…10b-pointer-tests Wave 6-P2-10b: Pointer layer v2 test coverage — 5 files, 86 cases
…11-runtime-gaps Wave 6-P2-11: runtime gap fixes — waitForProof audit + uxf-cid receive
…nt + refund soak
Closes the invoice-return refund path uncovered by wave 6-P2-9's
soak-invoice-return. Two changes:
## 1. UXF payload sender.directAddress wire hint
`extensions/uxf/types/uxf-transfer.ts` — `UxfTransferPayloadBase.sender`
gains an optional `directAddress` field. UNAUTHENTICATED on wire (same
as `nametag`), documented as a HINT the receiver uses to populate the
RECEIVED history entry's `senderAddress` without racing the sender's
Nostr identity binding publication. Security note: only affects the
caller's own money (refund destination), so a malicious sender only
harms themselves.
Sender path: `PaymentsModule.deliverTokens` +
`extensions/uxf/pipeline/module-glue/publish-uxf-bundle.ts` both
populate `sender.directAddress = identity.directAddress` on send.
Receiver path: `PaymentsModule.handleIncomingTransfer` prefers the
wire hint over `transport.resolveTransportPubkeyInfo` (wave 6-P2-6d
fallback). Eliminates the race that previously left `senderBalances[]`
empty when the sender's binding hadn't propagated yet.
## 2. Soak-invoice-return: use cancelInvoice(autoReturn) not
returnAllInvoicePayments
`scripts/soak-invoice-return.ts` restructured to test the correct
refund flow. Root cause discovered while investigating: on a fully-
COVERED/CLOSED invoice `returnAllInvoicePayments` returns 0 BY DESIGN
— the CLOSED freeze zeros per-sender balances via balance-computer.ts's
surplus-distribution algorithm. That API is for OVERPAID (surplus > 0)
invoices only.
The real refund flow for a normally-paid invoice is
`cancelInvoice({autoReturn: true})` — cancels the invoice AND refunds
all forwarded payments to their senders via the CANCELLED freeze path
which preserves balances.
The soak now: partial payment (500 of 1000) → invoice PARTIAL →
`cancelInvoice({autoReturn:true})` → B receives 500 UCT refund via
`transfer:incoming` → B's balance restored (5000 → 5000, delta 0).
## Validation
All 5 testnet2 soaks PASS end-to-end:
- soak-mint-throughput
- soak-two-wallet-send
- soak-multi-coin-bundle
- soak-invoice-lifecycle
- soak-invoice-return (new PASS state — was documented gap)
Typecheck 0 errors, test suite 7,205 pass / 6 skip preserved.
…12-sender-directaddress-hint Wave 6-P2-12: sender.directAddress wire hint + fix invoice-return soak
…shots Runs each of the 5 soaks N times (default 3) and reports per-soak wall-clock stats (min / p50 / mean / max) plus per-step counts. Baseline snapshot (2026-07-08, gateway.testnet2.unicity.network, 3 runs): mint-throughput (10 mints) 26.5s / p50 27.8s / max 30.8s two-wallet-send 13.4s / p50 14.1s / max 15.3s multi-coin-bundle 23.8s / p50 24.7s / max 26.3s invoice-lifecycle 16.2s / p50 16.3s / max 17.0s invoice-return 23.1s / p50 23.3s / max 25.3s Per-call medians extracted from log-step deltas: Sphere.init ~500 ms sphere.tokenEngine.mint ~2,741 ms (v2 atomic submit) sphere.accounting.createInvoice ~3,100 ms (data-token mint) sphere.payments.send ~8,400 ms (send + Nostr + IPFS pin) sphere.accounting.payInvoice ~8,500 ms (routed through send) transfer:incoming event fire ~500 ms No v1 baseline available (v1 aggregator retired pre-Phase-6), so this is a v2-only snapshot. Architecturally v2 should be faster: the migration removed v1's background commitment retry, finalization workers, and multi-shot proof polling — mint's 2.7s p50 is consistent with what an atomic-submit design achieves. Run: npx tsx scripts/soak-metrics.ts Prints a summary table to stderr + JSON blob to stdout for consumption by regression-tracking scripts.
…13-soak-metrics Wave 6-P2-13: soak-metrics harness for perf snapshots
Multiple Sphere.init calls within a process (CLI reruns, multi-wallet soaks, multi-tenant tests) each fetched the trust base JSON from raw.githubusercontent.com. That URL rate-limits at ~60 req/hr per IP, so multi-wallet workflows (e.g. trader-roundtrip with 4 wallets + tenants, or the soak-metrics harness's 15 runs) blow through the limit and Sphere.init throws. Fix: process-scoped `Map<url, parsedJson>` in `core/Sphere.ts` shared across all Sphere instances in the same process. First Sphere.init in a process fetches; subsequent instances re-use the parsed JSON. Zero behavior change for single-Sphere programs. Trust bases rarely change between soak runs on the same day; if a new validator set rotates in mid-process the cached value would stale, but that's acceptable for soak / CLI workflows (fix would be a restart) and matches how other tools handle trust-base rotation. Validated: facade smoke passes; rate-limit-triggered ensureTokenEngine throws no longer fire on the second+ Sphere.init.
…14-trust-base-cache Wave 6-P2-14: process-wide trust-base fetch cache
… nametag mints Fixes "Could not mint nametag token: Token engine not available" warnings that fire from background nametag-mint retries during Sphere.init / switchToAddress on slow networks. Three sites patched: - core/sphere-addresses.ts postSwitchSync (new-nametag arm + missing-token arm) — detached from switchToAddress, races the async trust-base fetch - core/Sphere.ts load-time nametag-token retry — same rationale Each site now `await sphere.ensureTokenEngine()` (idempotent) before calling mintNametag, so the wave 6-P2-4e wiring is fully settled by the time the background mint fires. Uncovered while running the trader-roundtrip soak against v2: multi- wallet init flows in sphere-cli were consistently hitting the race. Behavior change: none for warm-cache boots. On cold-start (trust base not yet fetched), background nametag mint waits for the fetch instead of failing silently and logging a warning. The observed symptom was 2 warnings per trader-roundtrip wallet.
…15-ensure-engine-before-mint Wave 6-P2-15: await ensureTokenEngine before background nametag mints
…egator API
The runtime `AggregatorClient` returned by `oracle.getAggregatorClient()`
is v2 post 6-P2-4a, but the pointer layer was still typed against v1 and
called `submitCommitment(requestId, transactionHash, authenticator)` —
which does not exist on v2, throwing `TypeError` on every publish. The
error was bucketed as `network_error` and burned through the retry
budget, so trader-roundtrip on v2 was spamming `Pointer publish failed
… AGGREGATOR_POINTER_NETWORK_ERROR`.
## Pointer commitment shape under v2
Wraps the pointer's synthetic per-side commit in a `PointerTransaction`
(new `pointer-transaction.ts`) that implements `ITransaction` with:
- `lockScript` = `EncodedPredicate.fromPredicate(SignaturePredicate.create(signingPubKey))`
- `sourceStateHash` = `DataHash(SHA256, deriveStateHashDigest(v, side))`
- `calculateTransactionHash()` = `DataHash(SHA256, ct)` — the pointer's
existing SPEC §6 ciphertext hash
Submit builds `SignaturePredicateUnlockScript.create(tx, signer)` which
signs `SHA256(sourceStateHash || transactionHash)`, then calls
`CertificationData.fromTransaction(tx, unlock)` and
`client.submitCertificationRequest(cert)`. Probe derives
`StateId.fromTransaction(tx)` and reads back via
`client.getInclusionProof(stateId)`; verification runs through
`InclusionProofVerificationRule.verify`, which requires a transaction —
we rebuild the pointer transaction with the received
`certificationData.transactionHash` so the rule can complete.
## v1 → v2 status enum mapping
| v1 `SubmitCommitmentStatus` | v2 `CertificationStatus` | Pointer bucket |
|----------------------------------------|-----------------------------------------|-----------------------|
| SUCCESS | SUCCESS | success |
| REQUEST_ID_EXISTS | (no direct equivalent — see below) | exists |
| AUTHENTICATOR_VERIFICATION_FAILED | SIGNATURE_VERIFICATION_FAILED | rejected (row 9) |
| — | INVALID_SIGNATURE_FORMAT | rejected (row 9) |
| — | INVALID_PUBLIC_KEY_FORMAT | rejected (row 9) |
| REQUEST_ID_MISMATCH | STATE_ID_MISMATCH | aggregator_rejected |
| — | INVALID_SOURCE_STATE_HASH_FORMAT | aggregator_rejected |
| — | INVALID_TRANSACTION_HASH_FORMAT | aggregator_rejected |
| — | UNSUPPORTED_ALGORITHM | aggregator_rejected |
| — | INVALID_SHARD | aggregator_rejected |
| unknown → protocol_error (fail closed) | unknown → exists (v2 tolerance contract)| exists |
v2 dropped the explicit "already exists" status — per
`CertificationResponse.d.ts` an unknown status means "the certification
for this state may already exist, probe getInclusionProof". The pointer
maps unknown statuses to `exists` so H2/H3 (marker match +
`isIdempotentRetryHint`) can distinguish idempotent replay (row 4) from
cross-device conflict (row 5) without adding a probe round-trip.
## Signing key derivation change
v2 dropped `SigningService.createFromSecret` — the constructor uses the
32-byte input directly AS the scalar. The pointer's `buildPointerSigner`
now calls `new SigningService(seed)`. Testnet2 wallets are fresh; there
is no wallet on both networks so the derived-pubkey change has no
migration surface. KAT vectors regenerated
(`tests/fixtures/pointer-kat-vectors.json`); the requestId vectors are
replaced with v2 StateId vectors.
## Files migrated
- `extensions/uxf/profile/aggregator-pointer/{signing,aggregator-submit,
aggregator-probe,publish-algorithm,discover-algorithm,
reconcile-algorithm,ProfilePointerLayer,trust-base-rotation,
win-broadcast}.ts` — imports switched from `stsdk-v1/*` to
`token-engine/sdk`; submit + probe rewritten on the v2 API.
- `extensions/uxf/profile/aggregator-pointer/pointer-transaction.ts` —
new synthetic `ITransaction` adapter.
- `extensions/uxf/profile/pointer-wiring.ts` — same import switch.
- `token-engine/sdk.ts` — added `InclusionProofVerificationRule` to the
barrel so the extension can verify inclusion proofs without breaking
the extension ESLint boundary.
Net LoC: +646 / -715 (≈70 lines smaller). All 7210 unit tests pass;
0 typecheck errors; 0 net new lint errors in the pointer files.
Refs #666 (Phase 6 UXF v2 SDK migration)
…16-pointer-v2-migration Wave 6-P2-16: migrate Profile pointer layer to v2 aggregator API
…bundles
Wave 6-P2-17 of the Phase-6 v1→v2 migration. Rewrites the Profile
token persistence pipeline to store v2 `SphereToken` blobs wrapped in
a canonical envelope (`SphereTokenPersistenceEntry`) — the same shape
the v2 wire path already ships over Nostr. Deletes the v1 chain
representation from all hot persistence / bundle paths. NO v1
backward compatibility, per the phase-6 mandate.
Root cause the wave fixes:
1. `PaymentsModule.tokenToTxfEntry` returned a v2 UI `Token` (no
`.genesis`) — the shape the v2 slim rebuild left behind.
2. `ProfileTokenStorageProvider.extractTokensFromTxfData` filtered
those entries out with "no genesis field" and handed an empty
token list to `UxfPackage.ingestAll`.
3. The published CAR carried zero tokens, so cross-process CLI
reads landed on empty state ("No tokens found").
Fixes:
* `types/txf.ts`: introduce `SphereTokenPersistenceEntry` (canonical
v2 envelope: `{_sdkVersion, _format, v, network, tokenId, token}`),
with a structural predicate for downstream shape checks. The v1
chain type names (`TxfToken`, `TxfGenesis`, `TxfState`,
`TxfTransaction`, `TxfIntegrity`, `TxfInclusionProof`,
`TxfAuthenticator`, `TxfMerkleTreePath`, `TxfMerkleStep`,
`TxfGenesisData`) survive as deprecated `any` aliases only —
just enough to keep the last v1-shape helpers (accounting
invoice payload constructor, `serialization/txf-serializer.ts`)
compiling until their own port.
* `modules/payments/PaymentsModule.ts`:
- `tokenToPersistenceEntry(...)` (replaces the old
`tokenToTxfEntry`) writes the v2 envelope, preferring live
re-encode from the `SphereToken` when the engine is present.
- `decodeSdkDataFromEntry(...)` restores the `SphereToken` from
the persisted envelope via the engine on `load()`.
- `loadFromStorageData(...)` gates on
`isSphereTokenPersistenceEntry` — non-v2 entries are logged
and skipped, not silently accepted.
* `extensions/uxf/profile/profile-token-storage-provider.ts`:
- `extractTokensFromTxfData(...)` recognizes the v2 envelope by
its shape signature. Non-v2 entries are logged and skipped.
* `extensions/uxf/bundle/UxfPackage.ts`: rewritten to a lean
v2-only class over a `Map<tokenId, SphereTokenPersistenceEntry>`.
- `toCar()` emits a single-block IPLD CARv1 whose dag-cbor root
block carries `{ version: 2, format: 'sphere-uxf-v2',
createdAt, updatedAt, tokens: [{tokenId, network, ver,
token}] }`. Entries are sorted by tokenId for deterministic
CIDs. The single-block CAR is fully compatible with the
existing `extractCarRootCid` / `pinCarBlocksToIpfs` pipeline.
- `fromCar()` parses the single root block, verifies the
`sphere-uxf-v2` format tag, and throws INVALID_PACKAGE on any
v1 shape.
- `merge()` is non-clobbering (`this` wins on tokenId collision)
to preserve the "keep the freshest local state" contract the
flush-scheduler relies on.
- Legacy methods that only made sense for the v1 element-pool /
instance-chain DAG (`verify`, `diff`, `applyDelta`,
`computeVerifiedProofs`, `consolidateProofs`, `addInstance`,
`filterTokens`, `tokensByCoinId`, `tokensByTokenType`, `gc`,
`save`, `toJson`) survive as trivial stubs for source compat.
* `extensions/uxf/profile/profile-token-storage/flush-scheduler.ts`:
the per-coin histogram now keys off `envelope.tokenId` (v2)
instead of the removed `genesis.data.coinData` field.
* `index.ts`: exports `SphereTokenPersistenceEntry` and
`isSphereTokenPersistenceEntry`; keeps the deprecated `TxfToken`
name for one wave of source-compat.
* `serialization/txf-serializer.ts`: two `any`/type widenings so
the v1 serializer (still used by the nametag store + accounting
legacy) keeps compiling under the loose `TxfToken = any` alias.
Test-suite hygiene:
Twenty-eight v1-shape tests (the UxfPackage internals suite, the
v1 conservative-/instant-sender pipelines, the v1 bundle-verifier
/ cid-fetcher / smuggled-roots suites, the byte-identical CAR
regression, the v1-shape profile flush / monotonicity / export /
manifest tests, and `txf-wire-roundtrip`) are moved to
`tests/legacy-v1/**` — the pre-existing v2-migration quarantine
that vitest already excludes.
Verification:
* `npx tsc --noEmit` — 0 errors.
* `npx vitest run` — 415 files, 6942 pass / 0 fail / 6 skip.
* All 5 testnet2 soaks (`mint-throughput`, `two-wallet-send`,
`multi-coin-bundle`, `invoice-lifecycle`, `invoice-return`) —
PASSED end-to-end.
* Cross-process CLI round-trip repro from the wave brief:
`sphere init` + `sphere faucet 100 UCT` in one process, then
a fresh `sphere status` + `sphere balance` process reads
`UCT: 100 (1 token)` off Profile persistence.
…17-profile-v2-persistence Wave 6-P2-17: v2-only Profile persistence via UXF bundles
…consumers
Wave 6-P2-18 of the Phase-6 v1→v2 SDK migration. Deletes the ten
deprecated v1 chain-shape name aliases that wave 6-P2-17 kept as
one-wave-of-source-compat `any` fallbacks (`TxfToken`, `TxfGenesis`,
`TxfGenesisData`, `TxfState`, `TxfTransaction`, `TxfIntegrity`,
`TxfInclusionProof`, `TxfAuthenticator`, `TxfMerkleTreePath`,
`TxfMerkleStep`) and migrates every non-legacy consumer.
Migration policy:
- v2-envelope-aware code (accounting invoice payload) reaches for
`SphereTokenPersistenceEntry`.
- v1-shape runtime code (still speaks state-transition-sdk v1 JSON
until the STSDK v2 swap lands) uses `unknown` at the type boundary
plus a local minimal interface for the fields it actually reads.
- No cross-file v1 alias re-use remains anywhere in the non-legacy
tree; each site's minimal interface is scoped to that file.
Files rewritten:
- `types/txf.ts`: aliases removed, docblock rewritten. `TxfStorageData`
index signature widened to `unknown` so adapters that still speak
v1 shape compile without the alias.
- `index.ts`: `TxfToken` name dropped from the public re-export.
- `modules/accounting/AccountingModule.ts` + `types.ts`: `createInvoice()`
now returns a canonical v2 `SphereTokenPersistenceEntry` envelope
(via `engine.encodeToken`). `importInvoice()` accepts the same
envelope shape, decodes via `engine.decodeToken`, reads invoice
bytes via `engine.readTokenData`, and cross-verifies the peer's
claimed tokenId against `engine.tokenId(sphereToken)`. Rejects a
value-bearing token. No v1 backward compat.
- `modules/accounting/types.ts`: `CreateInvoiceResult.token` retyped
to `SphereTokenPersistenceEntry`. `CoinEntry` docblock rewritten
off the deleted `TxfGenesisData.coinData` reference.
- `modules/payments/PaymentsModule.ts`: archived/forked token maps
retyped to `Map<string, unknown>`.
- `modules/payments/persistence/codec.ts`: archive host-slot types
widened to `unknown`; the single genesis-tokenId read narrowed via
a local `MinimalTxfShape`.
- `modules/payments/tokens/{identity,parse-cache,archive}.ts`:
local minimal interfaces replace the deleted aliases.
- `modules/payments/import-export/{import,export}.ts`: local
wire-shape interfaces replace the alias.
- `serialization/txf-serializer.ts`: public entry points
(`getCurrentStateHash`, `hasMissingNewStateHash`, `txfToToken`,
`buildTxfStorageData`, `parseTxfStorageData`) take `unknown` at the
boundary and narrow internally via a local `V1TxfShape`. Storage-
data option types widened to `Map<string, unknown>`.
- `extensions/uxf/profile/import-from-legacy.ts`: `LegacyTxfToken`
local interface.
- `extensions/uxf/profile/token-storage-migration.ts`: `MigrationTxfToken`
local interface.
Verification:
- `npx tsc --noEmit` — 0 errors.
…odeToken into fakes - Move three v1-shape token/serializer tests into `tests/legacy-v1/**` (already excluded from CI): `unit/profile/token-storage-migration`, `integration/profile/token-storage-migration`, and `unit/serialization/txf-serializer`. Each constructs `TxfToken` fixtures by hand; the shape they exercise is gone with the wave 6-P2-18 alias deletion. - Replace the `Map<string, TxfToken>` type in `tests/unit/modules/payments/PaymentsModule.importExport.v2.test.ts` with `Map<string, unknown>` — matches the widened archive / forked API surface. - Extend the fake tokenEngine in the two AccountingModule v2 tests (`AccountingModule.lifecycle.v2` and `AccountingModule.coinIdValidation.v2`) with an `encodeToken(sphereToken) → TokenBlob` stub so the new `SphereTokenPersistenceEntry` envelope wrap succeeds. - Prose cleanup on two `TxfToken` doc-comment references (`modules/swap/dm-protocol.ts`, `modules/payments/import-export/types.ts`) now that the alias is gone. Verification: `npx vitest run` — 412 files, 6849 pass / 0 fail / 6 skip.
…eted TxfToken Aligns the header comment with the wave-6-P2-18 wire contract: the soak's `importInvoice` step now speaks the v2 `SphereTokenPersistenceEntry` envelope, not the deleted v1 `TxfToken` shape.
…18-purge-v1-leaks Wave 6-P2-18: purge remaining v1 backward-compat leaks
Per the strict "No v1 backward compatibility!" directive, delete the last remaining v1 boundary — the legacy wallet import/migration flow — along with every symbol that only exists to serve it. Deleted files: - extensions/uxf/profile/import-from-legacy.ts (legacy wallet reader) - extensions/uxf/profile/token-storage-migration.ts (migrate helper) - modules/payments/import-export/ (whole directory: v1-shape parser and WireTxfToken local interface) - modules/payments/legacy-v1/PaymentsModule-legacy.ts (dead sibling that imported from the deleted import-export directory) Deleted from PaymentsModule (public API surface): - importTokens() method - ImportAddedCode, ImportSkipCode, ImportRejectCode, ImportAdded, ImportSkipped, ImportRejected, ImportTokensResult re-exports Deleted from extensions/uxf/profile/index.ts (public API surface): - ProfileMigration class - importLegacyTokens + LegacyImportOptions + LegacyImportResult - migrateTokenStorage / migrateLegacyToProfile / migrateProfileToLegacy + isTokenStorageMigrationComplete + clearTokenStorageMigrationMarker + TOKEN_STORAGE_MIGRATION_MARKER_VERSION - MigrationDirection / TokenStorageMigrationOptions / TokenStorageMigrationProgress / TokenStorageMigrationCounts / TokenStorageMigrationResult / MigrateLegacyToProfileOptions / MigrateLegacyToProfileFromSphereOptions / MigrateLegacyToProfileFromSphereResult Deleted from browser factory: - createBrowserProfileProvidersAuto (auto-wired legacy fallback) - migrateLegacyToProfileBrowser - BrowserProfileProviders.fallbackTokenStorage field Deleted from node factory: - migrateLegacyToProfileNode Retained (unrelated intra-profile schema migration): - migrateInvalidTokensToPerEntryKey (T.1.E; single-blob → per-entry-key `invalid.*`; no dependency on v1 wallet layout) Deleted from Sphere.load: - `migration.migratedAt` marker probe + Issue #330 warning path. The generic `fallbackTokenStorage` optional field survives on SphereInitOptions / SphereLoadOptions for callers that need to wire their own read-only fallback manually. Quarantined to tests/legacy-v1/ (wave 6-P2-5 pattern): - tests/unit/modules/payments/PaymentsModule.importExport.v2.test.ts - tests/unit/profile/token-storage-migration-from-sphere.test.ts - tests/unit/profile/migration.test.ts - tests/unit/profile/migration-c2-flush-before-cleanup.test.ts - tests/e2e/migrate-to-profile-conservation.test.ts Surgically trimmed: - tests/unit/profile/integration.test.ts — the `migration flow` describe block referenced ProfileMigration; the block is removed and the ProfileMigration import dropped. All other integration tests in the file (full lifecycle, multi-device, factory) are retained and continue to run. Net LoC delta: -13,234 (mostly deletions of PaymentsModule-legacy.ts and the two migration/import files). tsc --noEmit: 0 errors. Vitest suite: 408/408 files pass; 6794 tests pass, 6 skipped. Downstream break flagged for follow-up (not fixed in this PR): - sphere-cli's `wallet migrate` command in src/legacy/legacy-cli.ts imports `importLegacyTokens` from `@unicitylabs/sphere-sdk/profile`. When this SDK change lands, sphere-cli's `wallet migrate` will fail to build. Track separately.
…19-delete-legacy-import Wave 6-P2-19: delete legacy v1 import path
… transient errors
Wraps v2 SDK's AggregatorClient with a retrying decorator so
`submitCertificationRequest` transparently retries HTTP 5xx / 429 /
network errors with bounded exponential backoff (~13s total: 500ms +
1500ms + 3500ms + 7500ms). 4xx surfaces immediately (real client
errors).
Trigger: trader-roundtrip on v2 hit `Fatal startup error: Failed to
mint nametag token: Submit failed: {"status":503,...}` — the trader
tenant's Docker init fatally aborted on the first testnet2 gateway 503
during registerNametag. With this wrapper, mint/transfer/split all
tolerate transient gateway 503s and complete once the gateway
recovers.
- New non-pinned `token-engine/retrying-aggregator-client.ts` — duck-
typed proxy that wraps `submitCertificationRequest`. Everything else
(getInclusionProof, getLatestBlockNumber) passthrough — those have
their own polling / retry loops that would compound if double-
retried.
- Minimal edit to SHA-pinned `factory.ts`: change one line to construct
StateTransitionClient over the wrapped client. Re-sync note added at
the header.
Classifier mirrors `core/connectivity.ts::isTransientAggregatorError`
but embedded so token-engine doesn't drag connectivity.ts into the
engine bundle. Parses both `"status":503` (v2 JsonRpc shape) and
`HTTP 503` (URL-mode transports).
…20-retry-transient-agg-errors Wave 6-P2-20: retry submitCertificationRequest on transient aggregator errors
…ter createInvoice
`sphere.accounting.createInvoice()` returns the invoice as
{invoiceId, token, terms} but never registered it in
`payments.tokens`. In single-process code paths callers cache the
returned `token` and pass it directly to `importInvoice()`, so the
gap doesn't show. But cross-process modules (escrow-service) look up
the token later via `sphere.payments.getToken(invoiceId)` to deliver
its sdkData envelope to a payer — that returns undefined, so the
escrow's `getDepositInvoiceToken()` hits `deliver_deposit_invoice_no_token`
after five 500ms retries and the swap cancels with `SWAP_CANCELLED`.
Fix: after mint + envelope build, construct a v2 UI Token wrapper
(coinId='', amount='0' — invoice is a data token) with sdkData=envelope-JSON
and add it via `deps.payments.addToken()`. Now `payments.getToken(invoiceId)`
returns the sdkData envelope that escrow-service expects.
Non-fatal on registration failure (mint already succeeded; caller
still gets the token via return value).
Uncovered while running trader-roundtrip on v2 with a local
v2-rebuilt escrow tenant — escrow booted, negotiated, but couldn't
deliver deposit invoices.
…21-register-invoice-token Wave 6-P2-21: register invoice token in payments after createInvoice
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Draft review artifact for the Phase 6 v1 → v2 SDK migration on the
@vrogojin/uxf-v2fork branch. Not intended to merge — the fork intentionally diverges from mainstreammain. Opened purely to give reviewers a browsable diff.Final migration state (as of HEAD
8fa9f1ed)STSDK_CORE_BURNDOWNentriescore/Sphere.tsmodules/payments/PaymentsModule.tsmodules/accounting/AccountingModule.tsoracle/UnicityAggregatorProvider.ts~19k LoC v1 code quarantined; ~4.1k LoC of slim v2 modules replace it; ~5.7k LoC of Sphere concerns extracted into 9 sibling files.
Migration waves
Core migration (v1 → v2 SDK)
token-engine/sphere.tokenEngineTest migration + validation
Sphere.ts slim series
Follow-up waves
.env.example+soak-invoice-return.ts+ docsuxf-cidreceive wiring (+13 cases)sender.directAddresswire hint — fixes invoice-return refund path end-to-endRuntime validation
All 5 soak scripts PASS end-to-end against real
gateway.testnet2.unicity.network:soak-mint-throughputsoak-two-wallet-sendsphere.payments.send, verified viatransfer:incomingsoak-multi-coin-bundlesoak-invoice-lifecyclesphere.accounting.*soak-invoice-returncancelInvoice({autoReturn:true})→ refund lands + balance restoredBoth facade smoke scripts also pass:
scripts/smoke-testnet2-mint.ts—createSphereTokenEnginedirectscripts/smoke-testnet2-facade.ts—Sphere.init → sphere.tokenEngine.mintAnti-corruption layer
token-engine/is SHA-pinned from mainstreammain@ce758f6b. The v2 adapterSphereTokenEngine+createSphereTokenEnginefactory are exposed throughtoken-engine/index.ts; all v2 SDK access outsidetoken-engine/routes throughITokenEngine(enforced by ESLintno-restricted-imports).Wave 6-P2-12 note: sender.directAddress wire hint
Wave 6-P2-9's
soak-invoice-return.tsuncovered two intertwined issues:transport.resolveTransportPubkeyInfo(senderTransportPubkey)at receive time races the sender's Nostr identity binding publication. Result:senderAddressunresolved in RECEIVED history →senderBalances[]empty → refund path can't route.returnAllInvoicePaymentson a CLOSED invoice returns 0 BY DESIGN (CLOSED freeze zeros per-sender balances via balance-computer's surplus-distribution algorithm). That API is for OVERPAID invoices only.Wave 6-P2-12 fixes both:
UxfTransferPayloadBase.sender.directAddressfield added — UNAUTHENTICATED on wire (same asnametag), documented as a HINT the receiver uses to populatesenderAddressimmediately. Sender populates in bothPaymentsModule.deliverTokensandextensions/uxf/pipeline/module-glue/publish-uxf-bundle.ts. Receiver prefers the hint over the transport binding lookup (which stays as fallback for older peers).soak-invoice-return.tsrestructured to test the CORRECT refund flow: partial payment →cancelInvoice({autoReturn: true})→ verify refund lands + payer's balance restored.Security note: the hint only affects the caller's own money (refund destination), so a malicious sender only harms themselves.
Known follow-ups (all optional; none blocking)
PaymentsModule.importTokensis a documented no-op — only called by legacy TXF migration path (extensions/uxf/profile/import-from-legacy.ts); fresh v2 wallets never invoke it. Wave 6-P2-10a locks in the current behavior with a regression guard.init/create/load/importstatic factories) — those ARE the facade's remaining reason to exist. Natural stopping point already reached.tests/legacy-v1/; 88 highest-value cases restored via wave 6-P2-7 + 6-P2-10a/b, remainder is stretch coverage.