fix: self-heal manifest-unreferenced branch forks (stop wedged branches)

[ragnorc]

What & why

A first write to a table on a branch lazily forks it via Lance create_branch — a durable, two-phase op that advances Lance state before the atomic manifest publish — so a crash, deploy restart, or cancelled request future between the fork and the publish left a fully-formed branch ref the manifest never referenced, wedging every subsequent write to that table on that branch with "orphaned table state … incomplete prior delete; run cleanup" (and cleanup couldn't even fix it, since the branch was still live). This makes the per-table fork derived state of the manifest: the fork now returns a typed ForkOutcome and self-heals a manifest-unreferenced leftover (reclaim + re-fork under the write queue) on the next write across load/mutate/merge, with cleanup's reconciler broadened to a per-table authority test as the guaranteed backstop.

Backing issue / RFC

• Maintainer fix for a reproduced durability defect (interrupted branch fork wedges writes). No public issue/RFC; per the template note, maintainer internal process applies. Deterministic local repro added as the lead test commit.

Checklist

• Change is focused (one logical change) — the fork-reclaim fix; a small related stale-comment cleanup rides in its own chore: commit
• Tests added/updated for behavior changes — red→green regression (writes.rs), reconcile coverage (maintenance.rs), failpoints flip + preserved retryable test
• Public docs updated if user-facing surface changed (or N/A) — N/A user-facing; dev docs (writes.md, invariants.md) updated
• Reviewed against docs/dev/invariants.md — no Hard Invariant weakened; uses the authority-derived reconciler pattern the invariants prefer

Notes for reviewers

Reclaim is safe because branch first-writes now acquire the per-(table, branch) write queues up front (held through publish), so no in-process writer can be mid-fork; cross-process in-flight forks remain the already-documented one-winner-CAS gap (noted in invariants.md). Commits are file-granular and ordered red→green (3edbe2e is the RED test, 6e3204d the fix) since interactive hunk staging wasn't available. Only steady-state cost is added serialization of concurrent first-writes to the same (table, branch) — which already conflict at commit; main-branch and already-forked writes are unaffected.

---

Note

High Risk
Changes destructive fork reclaim, manifest authority checks, and write-queue ordering on load/mutate/merge/cleanup paths; mistakes could delete live forks or leave writes unserialized, though tests and failpoints target those cases.

Overview
Fixes a durability bug where an interrupted first-write fork (Lance create_branch before manifest publish) left a branch ref the manifest never referenced, wedging later writes with “run cleanup” even on live branches.

Write path: fork_branch_from_state now returns ForkOutcome::RefAlreadyExists instead of a hard error; the engine runs reclaim_orphaned_fork_and_refork after classify_fork_ref confirms the ref is an orphan from fresh manifest authority. Load and mutate acquire per-(table, branch) write queues up front for all touched tables (including cascade-delete edges) and pass held guards into commit_all so reclaim cannot race in-process forks and cannot self-deadlock on re-acquire.

Cleanup: reconcile_orphaned_branches treats orphans per table (live branch but table not placed on it), re-validates under the write queue via the same classifier, and skips on Indeterminate reads.

Dev docs and tests are updated (self-heal regressions, failpoint flips); minor comment/README tidy.

^{Reviewed by Cursor Bugbot for commit 1eea371. Bugbot is set up for automated code reviews on this repo. Configure here.}

Greptile Summary

This PR fixes a durability wedge on the branch first-write path: Lance's create_branch advances storage state before the manifest publish, so a crash or cancelled future left a fully-formed branch ref the manifest never referenced, causing every subsequent write to that table on that branch to error with "incomplete prior delete; run cleanup." The fix makes the fork a self-healing derived-state operation: fork_branch_from_state now returns a typed ForkOutcome rather than surfacing a collision as an error, reclaim_orphaned_fork_and_refork force-deletes and re-forks under fresh manifest authority, and reconcile_orphaned_branches is broadened to a per-table authority test (origin-2 orphans on live branches).

• Write path: mutation.rs and loader/mod.rs pre-acquire per-(table, branch) write queues for all touched tables (including cascade edges for node deletes) before the first fork and hold them through the publish; commit_all promotes the guard-coverage check from debug_assert to an all-builds error.
• Cleanup: reconcile_orphaned_branches now tests each (table, branch) pair against fresh manifest authority under the write queue via the shared classify_fork_ref classifier (Orphan / Legitimate / Indeterminate), preventing the stale-live_branches shortcut from destroying a legitimately-published fork on a newly-created branch.
• Tests: Red→green regression in writes.rs, per-table cleanup coverage in maintenance.rs, and failpoint tests pinning the Indeterminate skip, reconcile converge, and snapshot-failure caching paths in failpoints.rs.

Confidence Score: 5/5

Safe to merge. The destructive fork reclaim requires two independent gates to agree (in-process write queue serialization + fresh manifest authority via classify_fork_ref returning Orphan) before any force-delete runs; the Indeterminate path consistently skips rather than destroys on ambiguity. All previously flagged issues have been addressed.

The core fix is correct by construction: ForkOutcome replaces an opaque error with a typed signal, classify_fork_ref consolidates the 'safe to destroy?' decision into one site shared by both reclaim paths, and the write-queue pre-acquisition closes the in-process race. Previous review findings (debug_assert promotion, string-based error matching, Err(_) => destructive action, stale live_branches shortcut) are all addressed. The remaining inline comment concerns a rare diagnostic edge case in the Legitimate error arm where expected and actual versions could coincide — retryable and safe, just slightly misleading. Test coverage spans red→green regression, failpoint-injected Indeterminate/skip/converge paths, and the new per-table cleanup authority test.

table_ops.rs (reclaim_orphaned_fork_and_refork) — the Legitimate error arm's second fresh_snapshot_for_branch call can produce an expected==actual version mismatch if the read fails between classify and the version lookup; suggested fix inline.

Important Files Changed

Filename	Overview
crates/omnigraph/src/db/omnigraph/table_ops.rs	Core of the fix: adds ForkRefStatus classifier, classify_fork_ref (shared by both reclaim sites), and reclaim_orphaned_fork_and_refork; write path now calls db.fork_dataset_from_entry_state (self-healing wrapper) instead of the raw fork function
crates/omnigraph/src/db/omnigraph/optimize.rs	reconcile_orphaned_branches broadened to per-table authority: origin-2 orphans (manifest-unreferenced fork on a live branch) now detected via per-branch snapshot + re-validated under the write queue via classify_fork_ref before any force-delete
crates/omnigraph/src/exec/staging.rs	commit_all gains held_guards parameter for the fork path; the pre-held coverage check is promoted from debug_assert to an all-builds error, safely failing the write if the pre-computed touched set diverges from the actual staged/inline set
crates/omnigraph/src/exec/mutation.rs	mutate_as now lowers IR up-front (lower_named_mutation), computes touched_table_keys (including cascade edges for node deletes), pre-acquires per-(table, branch) write queues when any table needs forking, and threads the guards into commit_all
crates/omnigraph/src/loader/mod.rs	load_jsonl_reader gains the same up-front fork-queue acquisition pattern as the mutation path; when any touched table needs forking, guards for all touched tables are acquired before Phase 2 and threaded through commit_all
crates/omnigraph/src/storage_layer.rs	ForkOutcome added; fork_branch_from_state return type changed to Result<ForkOutcome>; RefAlreadyExists is now a typed signal (disambiguated via list_branches) rather than an opaque error
crates/omnigraph/src/table_store.rs	fork_branch_from_state disambiguates a failed create_branch: confirms via list_branches that the ref actually exists before returning RefAlreadyExists, propagating the raw error if the ref is absent (zombie or transient failure)
crates/omnigraph/tests/writes.rs	Two new regression tests: first_write_self_heals_manifest_unreferenced_fork_on_live_branch (via both load and mutate) and branch_cascade_delete_forks_node_and_edges_under_held_queues covering up-front queue coverage for cascade edge tables
crates/omnigraph/tests/failpoints.rs	Tests flipped from expecting wedge to expecting self-heal; new tests cover Indeterminate authority path (classify.fresh_read failpoint), reconcile skip-on-indeterminate then converge, and the caching test for live-branch snapshot resolution failure
crates/omnigraph/tests/maintenance.rs	New cleanup_reconciles_live_branch_orphan_fork_but_keeps_legitimate_fork test verifies origin-2 per-table authority: orphaned Person fork reclaimed while legitimately-forked Company on same live branch is preserved

Sequence Diagram

sequenceDiagram
    participant Caller as mutate_as / load_as
    participant WQ as WriteQueue
    participant OB as open_owned_dataset_for_branch_write
    participant FS as fork_dataset_from_entry_state (Omnigraph wrapper)
    participant TS as TableStore.fork_branch_from_state
    participant CFR as classify_fork_ref
    participant ROR as reclaim_orphaned_fork_and_refork
    participant CA as commit_all

    Caller->>WQ: acquire_many(touched tables x branch) [if needs_fork]
    WQ-->>Caller: guards held

    Caller->>OB: open for branch write
    OB->>FS: fork (source to active_branch)
    FS->>TS: create_branch(target_branch, source_version)

    alt RefAlreadyExists
        TS-->>FS: ForkOutcome::RefAlreadyExists
        FS->>ROR: reclaim_orphaned_fork_and_refork
        ROR->>CFR: classify_fork_ref(fresh authority)
        CFR-->>ROR: Orphan / Legitimate / Indeterminate
        alt Orphan
            ROR->>TS: force_delete_branch + re-fork
            ROR-->>FS: Ok(SnapshotHandle)
        else Legitimate / Indeterminate
            ROR-->>Caller: retryable error
        end
    else Created
        TS-->>FS: ForkOutcome::Created(dataset)
        FS-->>OB: Ok(SnapshotHandle)
    end

    OB-->>Caller: (dataset, Some(active_branch))
    Caller->>CA: commit_all(held_guards)
    CA->>CA: coverage check (queue_keys subset acquired_keys)
    CA-->>Caller: (updates, versions)
    Note over Caller,WQ: guards released at end of scope

Loading

Comments Outside Diff (1)

1. crates/omnigraph/src/exec/mutation.rs, line 456-469 (link)

   needs_fork = false leaves a fork-without-queue gap on cross-process branch recreate

   If needs_fork is false at snapshot time (all touched tables appear already-forked), no guards are pre-acquired. If a foreign-process then deletes and recreates the branch between this snapshot fetch and execute_named_mutation, some tables become unforked; open_owned_dataset_for_branch_write re-detects this and calls db.fork_dataset_from_entry_state, which may reach reclaim_orphaned_fork_and_refork — without the per-(table, branch) queue held. The invariant comment on reclaim_orphaned_fork_and_refork states callers "MUST already hold" the queue.

   This is acknowledged in invariants.md as the one-winner-CAS gap and leads to retries rather than corruption, so it's not a blocking concern. Worth calling out explicitly since the comment on reclaim_orphaned_fork_and_refork reads as an unconditional precondition but actually has this cross-process carve-out.

   

_{Reviews (13): Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." | Re-trigger Greptile}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 37f5af587a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d80cb12. Configure here.}