A graph whose search index cannot be reconciled is permanently unopenable

[vishaltandale00]

Problem

A desktop profile whose graph database survives an unclean shutdown can become
permanently unopenable. The graph server fails closed at startup, every launch,
with no way back to the data.

Error: open graph database

Caused by:
    0: reconcile the search index
    1: graph engine failed: rebuilt search store failed revision validation

Electron surfaces this as a native Relayer could not start dialog and quits.

The check is in crates/relayer-graph-server/src/search_index/lifecycle.rs.
open_reconciled rebuilds the Ladybug search store from a snapshot of the
canonical SQLite graph, then requires the rebuilt revisions to match:

if !revisions_match(&index, snapshot).await? {
    return Err(GraphError::Internal(
        "rebuilt search store failed revision validation".into(),
    ));
}

The failure is not corruption. On the affected database PRAGMA integrity_check
returns ok. The rebuild is driven from canonical SQLite, so a stale or missing
search store should be recoverable by definition — yet the rebuild still does not
reproduce the revisions SQLite declares.

Observed on

A Relayer Dev profile last written 2026-08-31, carrying a 655 KB
un-checkpointed graph.sqlite3-wal and a graph.sqlite3.ladybug store stamped
the same minute — the signature of an unclean shutdown. Reproduced against
relayer-graph-server built from a55c50dd; origin/main at 2f694047 changes
nothing under crates/relayer-graph-server/src/search_index/ or
crates/relayer-graph-core/, so 0.2.28 does not alter this path.

Reproduction

1. Copy an affected profile's graph database with its sidecars
   (graph.sqlite3, -wal, -shm, and the graph.sqlite3.ladybug directory).
   Copying the main file alone yields a misleading
   database disk image is malformed, which is an artifact of the incomplete
   copy, not the defect.
2. ./target/debug/relayer-graph-server --database <copy>/graph.sqlite3 --control-token <token> --port 0
3. The server exits with the error above.

What does not recover it

Each tried against an independent copy, with nothing else holding the database:

Attempt	Result
Delete graph.sqlite3.ladybug and let the store rebuild from scratch	still fails revision validation
PRAGMA wal_checkpoint(TRUNCATE) then open	still fails revision validation
Fresh empty database (control)	starts normally, {"ready":true,...}

The control confirms the binary is healthy. Discarding the graph is the only
known way forward, which means losing the user's threads and graphs.

Why this matters

docs/prd/index.html already requires the opposite of the current behavior:

If migration or startup fails, Relayer must fail closed. It must not change the
last recoverable database state. It must offer a recovery procedure.

Failing closed is correct and is happening. Not changing the last recoverable
state appears to hold — the Ladybug store is not overwritten on the failing path.
The missing half is the recovery procedure: today the user gets a modal dialog,
the app quits, and nothing in the product offers a way back.

Desired outcome

An existing graph whose search index cannot be reconciled is recoverable without
discarding the user's data, and the failure explains itself.

Acceptance criteria

• Establish why a rebuild driven from canonical SQLite produces revisions that
  disagree with SQLite. Name the divergence rather than widening the tolerance;
  a reconcile that cannot reproduce canonical revisions is either reading a state
  SQLite does not actually declare, or the snapshot is taken at a boundary that
  does not match what it validates against.
• Fix that divergence so an unclean-shutdown profile reconciles, or, where the
  state is genuinely unreconcilable, define and implement the recovery procedure
  the PRD requires: rebuild search from canonical SQLite as an explicit repair,
  and preserve the graph.
• The startup failure reports something a user can act on, not an internal
  message. It must remain fail-closed and must not mutate the last recoverable
  state.
• Deterministic coverage that opens a fixture graph in the unreconcilable state
  and proves the repair path restores service with the graph intact. The existing
  npm run check:graph-crash-reconciliation lane is the natural home.
• npm run check and npm run build pass on the final tree.

Notes

Found while running the desktop app in development during the customer-feedback
pass (#411-#418, PR #423). Unrelated to that PR, which contains no Rust. The
installed 0.2.25 app is unaffected and its profile was never touched; the dev
build uses a separate Relayer Dev profile via app.setName in
desktop/main/index.mjs.

Workaround for development in the meantime:
RELAYER_DESKTOP_USER_DATA_DIR=/tmp/relayer-demo npm run desktop:dev.

[vishaltandale00]

Root cause found

Not a data problem, and not recoverable by repairing the profile. The rebuilt
store this code writes disagrees with the canonical inventory this same code
computes, so the synchronous rebuild can never validate.

I instrumented revisions_match and tagged its three call sites. On the affected
database the failure is at the freshly rebuilt store:

DIAG callsite=BUILD_GENERATION(freshly rebuilt)
DIAG inventory mismatch: actual_len=119 expected_len=147
DIAG distinct: actual=119 expected=119
DIAG same-multiset=false same-order=false
DIAG count differs: actual=1 expected=2 entry="[[\"relationship\",\"CONNECTED\",\"edge:1\",\"content:2\",\"content:3\"], ...]"

Identical distinct sets (119 = 119). The two differ only in multiplicity, and
every differing entry is a CONNECTED relationship the canonical side expects
twice and the store holds once. 147 − 119 = 28, exactly the edge count.

The disagreement

canonical_inventory in crates/relayer-graph-server/src/search_index/schema.rs
emits each CONNECTED edge with multiplicity 2:

&targets,
// Ladybug 0.18 reports one undirected CONNECTED record in
// both orientations. An injected duplicate therefore adds
// two more entries to the physical multiset.
2,

physical_inventory, in the same file, reads them back with a directed
pattern and then normalizes endpoint order:

"MATCH (a)-[r:{kind}]->(b) RETURN r.id,a.id,b.id,{returned},r.published_targets"
...
if kind == "CONNECTED" && start > end {
    std::mem::swap(&mut start, &mut end);
}

A directed match returns one row per edge, and the swap makes the entry
orientation-independent, so the physical multiplicity is 1. The comment's
premise would hold only for an undirected pattern (-[r]-). The repository's own
crash test already encodes the physical truth:

("MATCH ()-[r:CONNECTED]->() RETURN count(r) AS count", "1"),

So the canonical expectation over-counts by exactly one per CONNECTED edge, and
build_generation's validation is unsatisfiable for any graph holding at least
one edge.

Introduced in a624fc8 (#337).

Why it is not already breaking every launch

open_reconciled reaches the synchronous rebuild only when the fast path is
unavailable. When an existing store opens and merely fails revisions_match,
control goes to damaged_targets, a background rebuild is spawned, and startup
returns Ok — the mismatch is invisible. The hard failure needs the fast path
skipped: store missing, unopenable, a legacy pointer, an unstable pointer, or a
component version change. An unclean shutdown produces exactly that.

Second consequence of the same defect: because damaged_targets compares the
same over-counted inventory per target, every target holding an edge is
classified damaged on every startup, so the background rebuild path is
entered perpetually even when nothing is wrong.

Candidate fix

Change the CONNECTED multiplicity from 2 to 1 and drop the stale comment.

Evidence for the change, all on this branch's tree:

• The previously unopenable database now opens: {"ready":true,"url":"http://127.0.0.1:61832"}.
• npm run check:graph-crash-reconciliation passes.
• cargo test --workspace passes, no failures.

I have not committed this. It is a one-line change in
crates/relayer-graph-server/src/search_index/schema.rs, and it wants a
regression test before it lands.

Missing coverage this exposes

No test combines the synchronous rebuild path with a graph containing a
CONNECTED edge. ladybug_crash_reconciliation.rs builds a graph with an edge
(edges: vec![edge.id]) and the lifecycle tests exercise rebuilds, but nothing
drives build_generation over an edge-bearing graph — which is why an
unsatisfiable validation shipped. That combination is the regression test.

The recovery-procedure gap in the issue body still stands on its own: even with
this fixed, an unreconcilable store should degrade to a repair rather than a
modal dialog and quit.