Fix corruption from rowid reuse when database transactions aren't committed

[ksuther]

I tried throwing Opus 4.8 and Fable 5 at our ever-persistent database corruption bug.

The first two commits are the primary fixes where we weren't handling rollbacks or failed commits properly and the key cache wasn't being invalidated.

The short version is the keyCache wouldn't get invalidated, so the mapping of collection/key to rowid wouldn't get reset. This could then lead to the wrong object getting saved at the wrong rowid because rowids get reused if the transaction isn't committed.

The other commits are other fixes that Fable found while analyzing the rollback bug.

Tests

• The rollback failure has a unit test (I don't think we were failing because of rollback).
• Fable simulated the commit failure by creating a small disk image, putting the database on the disk image, and filling up the disk image to trigger failing writes.

Giant Claude-generated description

Persistent on-disk corruption from a stale keyCache after an uncommitted write

Summary

Under specific conditions, YapDatabase can persist an object into the wrong row on disk — an object of one collection's type ends up stored in a row belonging to a different collection. Because the corruption is on disk,
an application that reads that row on every launch (for example, when a registered extension re-scans a collection at startup) crashes on every launch, producing a permanent boot loop.

Background

• Each YapDatabaseConnection keeps a keyCache: a bidirectional map between a row's integer rowid and its collection/key pair. Since the optimization that made keyCache authoritative, getRowid:forCollectionKey: (the write
  path) trusts the cache's reverse map and collectionKeyForRowid: (the read path) trusts the forward map, rather than resolving through the sqlite unique index.
• The primary table is declared rowid INTEGER PRIMARY KEY (no AUTOINCREMENT), so sqlite reuses freed rowids.
• On insert, the new rowid is written into keyCache eagerly, before the transaction's commit or rollback is decided.

The bug: two paths leave a phantom keyCache entry pointing at a rowid whose row was never actually committed. Combined with rowid reuse, a later operation then resolves that phantom to the wrong row.

Root cause — how a phantom entry is stranded

Path A — rollback after an insert. A read-write transaction inserts one or more rows (populating keyCache), then rolls back. On rollback the connection flushed objectCache and metadataCache but not keyCache, so the
eagerly-inserted rowid ⇄ collection/key entries survive for rows that no longer exist.

Path B — a failed COMMIT (no explicit rollback). A normal transaction inserts rows, then the COMMIT fails — e.g. SQLITE_FULL (disk full) or SQLITE_IOERR while writing the WAL at commit time. Per sqlite's documented
behavior, such an error can roll the entire transaction back automatically. YapDatabase logged the error but proceeded down the commit-success path: it kept the phantom keyCache entries, advanced the snapshot, and
broadcast the changeset to peer connections. No rollback flag was set, so the rollback cleanup never ran.

In both cases the phantom persists, and because inserts are broadcast to peer connections only by collection/key (never by rowid), peers never purge it either.

Reproduction sequence — durable corruption and a crash-on-every-launch loop

Consider an application that stores related data under the same key K in two collections — say collection A holds a model object, and a sibling collection B holds a plain dictionary for the same K (a common pattern for
storing side metadata next to a primary object).

1. Insert + eager cache. A transaction does setObject:aDictionary forKey:K inCollection:B. The insert obtains rowid = 500 from sqlite3_last_insert_rowid and immediately records 500 ⇄ (B, K) in keyCache.
2. The write is undone but the cache isn't. The transaction rolls back (Path A) or the COMMIT fails and sqlite auto-rolls-back (Path B). Nothing lands on disk — rowid 500 is free again — but keyCache still holds the
   phantom (B, K) → 500 (reverse) and 500 → (B, K) (forward).
3. Rowid reuse. Later, ordinary activity inserts a genuine object into collection A, and sqlite hands out the freed rowid 500. On disk, rowid 500 is now a valid A object.
4. The poisoned write hits disk. The app writes K's sibling data again: setObject:someDictionary forKey:K inCollection:B.
   - getRowid:forCollectionKey:(B, K) trusts the stale reverse map → returns rowid 500, with found = YES.
   - So it takes the UPDATE path: UPDATE ... SET data = WHERE rowid = 500.
   - The write-side type check passes, because it validates against the caller's collection (B, where a dictionary is a valid object) — it has no way to know rowid 500 actually belongs to collection A.

Result: the collection-A row at rowid 500 now contains an archived dictionary in its object column, on disk. The corruption is durable; it gets checkpointed out of the WAL into the main database file.
5. Every launch crashes. On the next launch, anything that reads collection A from a cold cache materializes rowid 500 through the object deserializer. If the deserializer (or the caller) expects the A type, it now
receives a dictionary and either asserts on the type mismatch or later sends an A-specific selector to the dictionary — e.g. unrecognized selector sent to instance on an __NSDictionaryI. If the reading code runs at startup
(for instance, an extension that re-scans the collection when it registers), the crash happens before the app is usable, on every relaunch — a permanent boot loop until the row is repaired or the database is rebuilt.

Note there is also a purely in-memory variant of the same defect: within one running process, the phantom forward entry makes a secondary-index enumeration resolve a reused rowid to the wrong collection/key, and
objectForCollectionKey: returns the wrong cached object straight from objectCache (no disk read, no deserializer, no assert). That variant is transient and disappears when the process restarts; the on-disk variant above is
the one that survives relaunch.

Fix

Purge keyCache on both paths that can strand a phantom entry:

• On rollback, flush keyCache alongside objectCache/metadataCache (and extension caches) in the connection's rollback cleanup.
• On a failed COMMIT, detect the failure (check sqlite3_get_autocommit() and the COMMIT return code), treat it exactly like a rollback — flush all caches, roll back extension state and memory tables, undo the snapshot
  increment, and retract the pending changeset instead of broadcasting it, so peer connections never observe the failed transaction.

With either fix in place, getRowid:forCollectionKey: and collectionKeyForRowid: never resolve a reused rowid to a stale collection/key, so nothing is written to the wrong row and there is no launch-time crash loop.

Two defense-in-depth measures also help contain an already-corrupt file: ensuring sibling rows can't be orphaned and have their rowids reused, and removing stale -wal/-shm sidecar files during corruption recovery so a
rebuilt database can't replay the old frames.