ci / vfs / doc: inconsistency fixes

Author: ditalini

.github/workflows/ci.yml

@@ -140,10 +140,13 @@ jobs:
           CGO_ENABLED: '0'
         run: |
           go build ./
-          go build ./gorm/...
           go build ./fts/...
           go build ./vfs/...
           go build ./vec/... || echo "vec not supported on ${GOOS}/${GOARCH} — skipping"
+          # gorm is a separate module — cross-build its dialector from its own
+          # dir. Build the package (`./`), not `./...`: gorm/examples/vec imports
+          # gosqlite.org/vec, which isn't transpiled on every target.
+          (cd gorm && go build ./)
 
   # Run the SQL conformance suite under tests/sql/. This is our methodical
   # feature-by-feature proof that every documented SQLite SQL surface the

README.md

@@ -141,6 +141,8 @@ Plain `database/sql` works too: `sql.Open("sqlite", "file:app.db")`. The full se
 
 Full per-package recipes, the `_*` DSN-flag table, and build-tag mapping: **[Migrating](docs/guides/migrating.md)**, **[DSN flags](docs/reference/dsn-flags.md)**, **[Build tags](docs/reference/build-tags.md)**. Runnable: [`examples/migrating/`](examples/migrating/).
 
+The drop-in claim is **CI-enforced**: every push runs `gorm.io/gorm`'s full integration suite, plus vendored subsets of `mattn/go-sqlite3`'s and `modernc.org/sqlite`'s own test suites, against this module (with an xorm-compatibility lane alongside) — so compatibility is checked by tests upstream wrote, not by tests we wrote to flatter ourselves. Recipes and divergence tables: [`dev/upstream/`](dev/upstream/).
+
 ## Why CGo-free
 
 Because SQLite is transpiled to Go (via `modernc.org/sqlite`) rather than C-bound, you get: builds in `golang:alpine` / distroless with no `apk add`; `GOOS=… GOARCH=… go build` cross-compilation that just works; clean `go test -race`; reproducible builds with no vendored amalgamation; and CI on providers that disallow CGo. The cost is a constant-factor gap on hot UDF/per-row paths — invisible for most applications. More in [Getting started](docs/getting-started.md#why-cgo-free) and [Performance](docs/reference/performance.md).

dev/upstream/gorm.md

@@ -122,14 +122,17 @@ ergonomics every user needs.
 This is what allowed the suite to flip from 1 fail → 0 fail on the
 second local run.
 
-## Why we run this manually rather than in CI today
-
-The setup churns ~20 indirect modules (postgres, mysql, sqlserver, mssql,
-gaussdb drivers — even though we only exercise SQLite, the gorm/tests
-module brings them all in). The CI matrix would slow down by 60–90s per
-job. We may add a single-job gorm-upstream lane later; for now, treat
-this matrix as a periodic check, run on dependency bumps and before
-release tags.
+## How this runs in CI
+
+This is a dedicated `gorm-upstream` job in `.github/workflows/ci.yml`,
+on every push and pull request to `main`. It's a separate job on
+purpose: the `gorm/tests` module pulls in ~20 indirect drivers
+(postgres, mysql, sqlserver, mssql, gaussdb) even though we only
+exercise SQLite, which adds 60–90s — isolating it keeps the main test
+matrix fast while still gating merges on the suite. The job pins
+`GORM_VERSION`, wires the shim above, runs the full `gorm/tests`, fails
+on any `--- FAIL`, and also fails if the `--- PASS` count regresses
+below its expected floor.
 
 When SQLite or gorm bumps, the steps to repeat:
 

docs/guides/custom-vfs.md

@@ -19,13 +19,15 @@ db.Close()
 vfs.Unregister("myvfs")              // after every db against it is closed
 ```
 
-Implement [`vfs.VFS`](https://pkg.go.dev/gosqlite.org/vfs#VFS) (`Open`/`Delete`/`Access`/`FullPathname`) and [`vfs.File`](https://pkg.go.dev/gosqlite.org/vfs#File) (`ReadAt`/`WriteAt`/`Truncate`/`Sync`/`Size`/locking/`Close`). Embed `vfs.NoLock` to satisfy the advisory-lock trio with accept-everything semantics (correct for single-process backends). Return a `vfs.VFSError` from any method to surface a specific `SQLITE_*` result code; a plain error becomes `SQLITE_IOERR`.
+Implement [`vfs.VFS`](https://pkg.go.dev/gosqlite.org/vfs#VFS) (`Open`/`Delete`/`Access`/`FullPathname`) and [`vfs.File`](https://pkg.go.dev/gosqlite.org/vfs#File) (`ReadAt`/`WriteAt`/`Truncate`/`Sync`/`Size`/locking/`Close`). Embed `vfs.NoLock` to satisfy the advisory-lock trio with accept-everything semantics — correct only for **single-connection** access (multiple connections in WAL mode need real locking; see [WAL](#wal--the-shmfile-capability)). Return a `vfs.VFSError` from any method to surface a specific `SQLITE_*` result code; a plain error becomes `SQLITE_IOERR`.
 
 A complete ~80-line in-memory backend is at [`examples/features/vfs/custom/`](../../examples/features/vfs/custom/main.go).
 
 ## WAL — the ShmFile capability
 
-A custom VFS runs in rollback-journal mode by default. To unlock WAL, have your `File` also implement `vfs.ShmFile` — a single `ShmGroup() string` method declaring which open files share a WAL index. The dispatcher owns the shared memory and the 8-slot WAL lock table, so you never touch unsafe memory or the lock protocol. WAL coordination is in-process (it backs multiple `database/sql` connections to one Go-managed database within a process, not cross-process WAL over a real disk).
+A custom VFS runs in rollback-journal mode by default. To unlock WAL, have your `File` also implement `vfs.ShmFile` — a single `ShmGroup() string` method declaring which open files share a WAL index. The dispatcher owns the shared memory and the 8-slot WAL lock table, so you never touch unsafe memory or the shared-memory lock protocol. WAL coordination is in-process (it backs multiple `database/sql` connections to one Go-managed database within a process, not cross-process WAL over a real disk).
+
+> **Multi-connection WAL needs real db-file locking — do not embed `vfs.NoLock`.** The dispatcher arbitrates the WAL *shared-memory* locks, but SQLite still gates destructive operations — notably the checkpoint it runs when a connection closes, which resets the `-wal` — on first acquiring an EXCLUSIVE *db-file* lock. `vfs.NoLock` grants that EXCLUSIVE even while other connections are active, so the close-checkpoint can reset the WAL under a concurrent writer and corrupt the database. Implement real `Lock` / `Unlock` / `CheckReservedLock` on the main db file: many connections may share `LockShared`; `LockExclusive` must fail while any other connection holds `LockShared`. The reference `File` in the `vfs` package tests is a complete in-process example.
 
 ## Instrumentation — Wrap
 

justfile

@@ -161,7 +161,8 @@ cross-build:
         export GOOS=$(echo "$triple" | cut -d/ -f1); \
         export GOARCH=$(echo "$triple" | cut -d/ -f2); \
         printf "  %-18s " "$triple"; \
-        go build ./ ./gorm/... ./fts/... ./vfs/... 2>/dev/null && \
+        go build ./ ./fts/... ./vfs/... 2>/dev/null && \
+        (cd gorm && go build ./ 2>/dev/null) && \
         (go build ./vec/... 2>/dev/null || echo -n "(vec skipped) ") && \
         echo "ok" || echo "FAILED"; \
     done

skills/custom-vfs/SKILL.md

@@ -34,13 +34,15 @@ type File interface {
 }
 ```
 
-- **Embed `vfs.NoLock`** in your File to get the lock trio for free (correct for single-process backends).
+- **Embed `vfs.NoLock`** in your File to get the lock trio for free — correct only for **single-connection** access (see the WAL caveat below; multi-connection WAL needs real locking).
 - **Errors:** return `&vfs.VFSError{Code: sqlite3.SQLITE_READONLY}` (or any `SQLITE_*`) for a specific code; a plain error becomes `SQLITE_IOERR`. A short read past EOF returns `io.EOF` (dispatcher zero-fills + reports SHORT_READ).
 - **Buffers are copied at the boundary** — a `File.ReadAt` is handed a fresh slice; don't alias it past the call.
 
 ## WAL
 
-Default is rollback-journal. For WAL, also implement `vfs.ShmFile` (one method, `ShmGroup() string`, declaring which files share a WAL index). The dispatcher owns the shared memory + lock table. In-process only.
+Default is rollback-journal. For WAL, also implement `vfs.ShmFile` (one method, `ShmGroup() string`, declaring which files share a WAL index). The dispatcher owns the shared memory + WAL lock table. In-process only.
+
+> **Multi-connection WAL needs real db-file locking — do NOT embed `vfs.NoLock`.** The dispatcher arbitrates the WAL *shared-memory* locks, but SQLite still gates destructive operations — notably the checkpoint it runs when a connection closes, which resets the `-wal` — on first acquiring an EXCLUSIVE *db-file* lock. `NoLock` hands out that EXCLUSIVE even while other connections are active, so the close-checkpoint can reset the WAL under a concurrent writer and corrupt the database. Implement real `Lock` / `Unlock` / `CheckReservedLock` on the main db file: many holders may share `LockShared`; `LockExclusive` must fail while any other connection holds `LockShared`. The reference `File` in the `vfs` package tests shows the full pattern.
 
 ## Instrumentation
 

vfs/interface.go

@@ -80,7 +80,8 @@ type File interface {
 	Size() (int64, error)
 
 	// Lock raises the advisory lock to at least level; Unlock lowers it.
-	// A single-process backend can accept every transition (see NoLock).
+	// A single-connection backend can accept every transition (see NoLock);
+	// multiple connections in WAL mode require real arbitration here.
 	Lock(level LockLevel) error
 	Unlock(level LockLevel) error
 	// CheckReservedLock reports whether some connection (possibly
@@ -203,8 +204,19 @@ const (
 )
 
 // NoLock is an embeddable helper supplying accept-everything advisory
-// locking — the correct behaviour for any single-process or
-// exclusive-access File. Embed it to drop three methods of boilerplate:
+// locking. It is correct only when a database is reached by ONE connection at
+// a time — single-connection use, or exclusive-locking mode — where there is
+// nothing to arbitrate and every lock can be granted.
+//
+// Do NOT embed NoLock when several connections share a database in WAL mode.
+// SQLite gates destructive operations — notably the checkpoint it runs when a
+// connection closes, which resets the -wal file — on first acquiring an
+// EXCLUSIVE db-file lock; NoLock hands out that EXCLUSIVE even while other
+// connections are active, so the reset can corrupt the database under a
+// concurrent writer. A multi-connection backend must implement real
+// Lock/Unlock/CheckReservedLock (many holders may share SHARED; EXCLUSIVE must
+// fail while any other connection holds SHARED) — see the reference File in the
+// vfs package tests.
 //
 //	type myFile struct {
 //		vfs.NoLock

vfs/interface_test.go

@@ -28,6 +28,15 @@ type refMemVFS struct {
 type refMemData struct {
 	mu   sync.Mutex
 	data []byte
+
+	// In-process advisory lock state, shared by every connection that opens
+	// this name. NoLock is fine for a single connection, but multi-connection
+	// WAL needs a real lock: SQLite gates its destructive checkpoint-on-close
+	// (which resets the -wal) on acquiring an EXCLUSIVE db-file lock, so that
+	// EXCLUSIVE must fail while other connections hold SHARED.
+	lmu     sync.Mutex
+	nShared int         // connections holding >= SHARED
+	writer  *refMemFile // the one connection holding RESERVED..EXCLUSIVE (nil if none)
 }
 
 func newRefMemVFS() *refMemVFS { return &refMemVFS{files: map[string]*refMemData{}} }
@@ -70,10 +79,80 @@ func (v *refMemVFS) Access(name string, _ vfs.AccessOp) (bool, error) {
 func (v *refMemVFS) FullPathname(name string) (string, error) { return name, nil }
 
 type refMemFile struct {
-	vfs.NoLock // single-process backend: accept every advisory lock
-	v          *refMemVFS
-	name       string
-	d          *refMemData
+	v    *refMemVFS
+	name string
+	d    *refMemData
+	lock vfs.LockLevel // this connection's current advisory lock level
+}
+
+var errLockBusy = &vfs.VFSError{Code: sqlite3.SQLITE_BUSY}
+
+// Lock raises this connection's advisory lock toward level, arbitrating in
+// process against the other connections on the same name. Multiple holders may
+// share SHARED; only one may hold RESERVED..EXCLUSIVE; EXCLUSIVE additionally
+// requires that no other connection holds SHARED.
+func (f *refMemFile) Lock(level vfs.LockLevel) error {
+	if level <= f.lock {
+		return nil
+	}
+	d := f.d
+	d.lmu.Lock()
+	defer d.lmu.Unlock()
+	switch level {
+	case vfs.LockShared:
+		if d.writer != nil && d.writer.lock >= vfs.LockPending {
+			return errLockBusy // a PENDING/EXCLUSIVE writer blocks new readers
+		}
+		d.nShared++
+		f.lock = vfs.LockShared
+	case vfs.LockReserved:
+		if d.writer != nil && d.writer != f {
+			return errLockBusy
+		}
+		d.writer = f
+		f.lock = vfs.LockReserved
+	case vfs.LockPending, vfs.LockExclusive:
+		if d.writer != nil && d.writer != f {
+			return errLockBusy
+		}
+		d.writer = f
+		self := 0
+		if f.lock >= vfs.LockShared {
+			self = 1
+		}
+		if d.nShared > self {
+			f.lock = vfs.LockPending // hold the intent so no new SHARED is granted
+			return errLockBusy
+		}
+		f.lock = level
+	}
+	return nil
+}
+
+// Unlock lowers this connection's advisory lock toward level.
+func (f *refMemFile) Unlock(level vfs.LockLevel) error {
+	if level >= f.lock {
+		return nil
+	}
+	d := f.d
+	d.lmu.Lock()
+	defer d.lmu.Unlock()
+	if f.lock >= vfs.LockReserved && level < vfs.LockReserved && d.writer == f {
+		d.writer = nil
+	}
+	if f.lock >= vfs.LockShared && level < vfs.LockShared {
+		d.nShared--
+	}
+	f.lock = level
+	return nil
+}
+
+// CheckReservedLock reports whether some connection holds RESERVED or higher.
+func (f *refMemFile) CheckReservedLock() (bool, error) {
+	d := f.d
+	d.lmu.Lock()
+	defer d.lmu.Unlock()
+	return d.writer != nil && d.writer.lock >= vfs.LockReserved, nil
 }
 
 func (f *refMemFile) ReadAt(p []byte, off int64) (int, error) {
@@ -127,7 +206,10 @@ func (f *refMemFile) Size() (int64, error) {
 
 func (f *refMemFile) SectorSize() int                        { return 512 }
 func (f *refMemFile) DeviceCharacteristics() vfs.DeviceFlags { return 0 }
-func (f *refMemFile) Close() error                           { return nil }
+func (f *refMemFile) Close() error {
+	_ = f.Unlock(vfs.LockNone)
+	return nil
+}
 
 // compile-time proof the reference types satisfy the public interfaces.
 var (
@@ -157,8 +239,7 @@ func TestUserVFS_ReadWrite(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	// Single connection: NoLock means no cross-conn arbitration, and the
-	// shared in-memory backing must not see concurrent writers.
+	// Single connection keeps this rollback-journal smoke test simple.
 	db.SetMaxOpenConns(1)
 
 	t.Cleanup(func() {