vec/gorm + fts/gorm: wrapper type, DropTable cascade, FTS5 modes,

dedupe helpers

Author: arthurpro

README.md

@@ -142,6 +142,74 @@ matches, _ := idx.SearchSlice(ctx, fts.Term("fox"),
 
 See [`examples/fts-search/`](examples/fts-search/main.go).
 
+### Deep gorm integration — tag-driven vec & FTS5
+
+The `vec/gorm` and `fts/gorm` sub-packages bridge gorm models to the
+vector / full-text sidecars. Tag a field, register the plugin, and
+gorm Create/Update/Delete maintains the sidecar automatically. Typed
+`KNN[T]` / `Search[T]` helpers return matching gorm models in
+ranking order with distance / rank attached.
+
+```go
+import (
+    _ "github.com/go-again/sqlite"
+    sqlitegorm "github.com/go-again/sqlite/gorm"
+    "github.com/go-again/sqlite/fts"
+    ftsgorm "github.com/go-again/sqlite/fts/gorm"
+    vecgorm "github.com/go-again/sqlite/vec/gorm"
+)
+
+type Document struct {
+    ID        uint   `gorm:"primaryKey"`
+    Title     string `fts5:"tokenize=porter+unicode61"`
+    Body      string `fts5:"tokenize=porter+unicode61"`
+    Embedding vecgorm.Embedding `vec:"dim=384;metric=cosine"`
+}
+
+db, _ := gorm.Open(sqlitegorm.Open("app.db"), &gorm.Config{})
+db.Use(vecgorm.Plugin())
+db.Use(ftsgorm.Plugin())
+
+vecgorm.Migrate(db, &Document{}) // creates documents + documents_vec
+ftsgorm.Migrate(db, &Document{}) // creates documents_fts + triggers
+
+db.Create(&Document{Title: "Hello", Body: "world", Embedding: vec})
+
+// Find documents semantically similar to a query vector:
+near, _ := vecgorm.KNN[Document](ctx, db, queryVec, 5)
+
+// Find documents matching a phrase, ranked by BM25:
+hits, _ := ftsgorm.Search[Document](ctx, db, fts.Term("world"))
+```
+
+Tag-driven features:
+
+- Auto-migrate sidecar tables alongside `db.AutoMigrate`.
+- Sync-on-write callbacks (vec) or triggers (FTS5) — including
+  `CreateInBatches` in a single transaction per batch.
+- Soft-delete awareness: models using `gorm.DeletedAt` get a
+  metadata column on the sidecar; KNN/Search excludes them
+  automatically. Pass `IncludeDeleted()` to override.
+- Typed helpers return `[]Result[T]` ordered by ranking so callers
+  don't have to rebuild the IN-clause + re-sort dance.
+- `db.Migrator().DropTable(&Model{})` cascades into the sidecar
+  (vec0 table or FTS5 table + triggers) via the dialector's
+  `DropTableHook` interface — no manual cleanup needed.
+- FTS5 mode is configurable per tag: `external` (default,
+  triggers-driven), `external=false` (in-table FTS5), or
+  `contentless=true` (index only, no text).
+
+The embedding field type is `vecgorm.Embedding` (a `[]float32`
+alias that implements gorm's `GormDataType`); `[]float32` with
+`gorm:"-"` also works for callers who prefer not to import the
+wrapper.
+
+See [`vec/gorm/`](vec/gorm/) and [`fts/gorm/`](fts/gorm/) for full
+package docs and [`examples/gorm-vec-tagged/`](examples/gorm-vec-tagged/)
++ [`examples/gorm-fts-tagged/`](examples/gorm-fts-tagged/) for
+runnable end-to-end usage; coverage matrix lives in
+[`docs/coverage-gorm.md`](docs/coverage-gorm.md).
+
 ### `embed.FS`-backed read-only databases
 
 ```go

docs/coverage-gorm.md

@@ -166,3 +166,68 @@ result is **386 PASS / 0 FAIL / 22 SKIP**. The setup, reproduction
 recipe, and reasoning for each shim flag live in
 [gorm-upstream.md](gorm-upstream.md). The same recipe is enforced by
 the `gorm-upstream` CI job.
+
+## Deep integration: `vec/gorm` and `fts/gorm`
+
+Tag-driven sidecar packages live under `github.com/go-again/sqlite/vec/gorm`
+and `github.com/go-again/sqlite/fts/gorm`. They register as gorm
+plugins and own the full lifecycle of the sidecar (vec0 virtual table /
+FTS5 external-content table + triggers).
+
+### Tag syntax — vec
+
+| Key | Required | Meaning |
+|---|---|---|
+| `dim=N` | yes | Embedding dimension. |
+| `metric=l2 \| cosine \| dot` | no | Distance metric. Default `l2`. |
+| `encoding=json \| binary` | no | Wire encoding. Default `binary`. |
+| `table=NAME` | no | Override sidecar table name. Default `<source>_vec`. |
+| `column=NAME` | no | Override embedding column. Default `embedding`. |
+
+The tagged field's type must be either `vecgorm.Embedding`
+(recommended) or `[]float32` with `gorm:"-"` alongside. The wrapper
+type implements gorm's `GormDataType` interface so the schema parser
+accepts it; the plugin then sets `IgnoreMigration=true` so no column
+lands on the source table.
+
+### Tag syntax — fts5
+
+| Key | Required | Meaning |
+|---|---|---|
+| `tokenize=NAME[+args]` | no | FTS5 tokenize option. Spaces escaped as `+`. |
+| `prefix=N1,N2,...` | no | Pre-computed prefix-match index sizes. |
+| `column=NAME` | no | Override FTS5 column name (default = lowercase field). |
+| `table=NAME` | no | Override FTS5 table. Default `<source>_fts`. |
+| `detail=full \| column \| none` | no | FTS5 detail= option. |
+| `external=true \| false` | no | External-content mode (default true). false → in-table FTS5 manages text itself. |
+| `contentless=true` | no | Contentless FTS5 (index only, no text). Snippet/highlight are rejected at search time. Mutually exclusive with `external=true`. |
+
+Multiple `fts5:`-tagged fields on one model share **one** FTS5 table.
+Conflicting table-level keys across fields are rejected at parse time.
+
+### Lifecycle matrix
+
+| Event | vec/gorm behavior | fts/gorm behavior |
+|---|---|---|
+| Plugin install | `db.Use(vecgorm.Plugin())` | `db.Use(ftsgorm.Plugin())` |
+| AutoMigrate | `vecgorm.Migrate(db, &T{})` creates source + sidecar | `ftsgorm.Migrate(db, &T{})` creates source + FTS5 table + triggers |
+| Create | AfterCreate callback `BatchInsert` (single tx) | AFTER INSERT trigger writes to FTS5 |
+| Save/Update | AfterUpdate callback `(*vec.Table).Update` | AFTER UPDATE trigger refreshes index |
+| Delete (hard) | AfterDelete callback `(*vec.Table).Delete` | AFTER DELETE trigger emits FTS5 `'delete'` |
+| Delete (soft, via `gorm.DeletedAt`) | Sidecar `deleted` flag flipped to 1 | FTS5's UNINDEXED `deleted_at` mirror set by trigger |
+| KNN / Search | Soft-deleted excluded by default; `IncludeDeleted()` overrides | Same |
+| DropSidecar | Drops sidecar table | Drops FTS5 table + all three triggers (for external mode) |
+| Source DropTable | Cascades into sidecar via DropTableHook on our gorm Dialector | Cascades into FTS5 table + triggers |
+| dim mismatch on re-migrate | Logged warning, existing sidecar left alone | n/a |
+
+### Tests
+
+| File | Tests | Notes |
+|---|---|---|
+| `vec/gorm/vecgorm_test.go` | 12 | Basic create/update/delete, KNN ranking, BatchInsert single-tx, soft-delete, Embedding wrapper |
+| `vec/gorm/lifecycle_test.go` | 9 | DropTable cascade, DropSidecar, composite PK rejection, tag validation, WithFilter, dim mismatch |
+| `fts/gorm/ftsgorm_test.go` | 10 | Migrate creates index + triggers, search/snippet/highlight, ranking, soft-delete, backfill |
+| `fts/gorm/lifecycle_test.go` | 8 | Conflicting tags, non-string fields, composite PK, LIMIT/OFFSET, no-plugin error, DropTable cascade |
+| `fts/gorm/mode_test.go` | 7 | external/in-table/contentless modes, conflicting modes rejected, contentless rejects snippet, in-table soft-delete |
+
+46 tests total, all passing on linux/macos.

examples/gorm-fts-tagged/main.go

@@ -0,0 +1,97 @@
+// gorm-fts-tagged: the tag-driven flow for combining gorm models with
+// SQLite FTS5. Mark text fields with `fts5:"..."` on a model, register
+// the ftsgorm.Plugin(), call ftsgorm.Migrate (which creates the FTS5
+// external-content table + the AFTER INSERT/UPDATE/DELETE triggers),
+// and let db.Create / ftsgorm.Search handle the index transparently.
+//
+// Compare with examples/gorm-fts/ for the side-by-side recipe — both
+// patterns coexist; pick the one that fits.
+package main
+
+import (
+	"context"
+	"fmt"
+	"log"
+
+	"gorm.io/gorm"
+	"gorm.io/gorm/logger"
+
+	_ "github.com/go-again/sqlite"
+	"github.com/go-again/sqlite/fts"
+	ftsgorm "github.com/go-again/sqlite/fts/gorm"
+	sqlitegorm "github.com/go-again/sqlite/gorm"
+)
+
+// Article is a typical gorm model with two indexed text fields. Both
+// are tagged with `fts5:` — they share one FTS5 external-content
+// table named `articles_fts` (the default `<source>_fts` form).
+type Article struct {
+	ID    uint   `gorm:"primaryKey"`
+	Title string `fts5:"tokenize=porter+unicode61"`
+	Body  string `fts5:"tokenize=porter+unicode61"`
+}
+
+func main() {
+	db, err := gorm.Open(sqlitegorm.Open(":memory:"), &gorm.Config{
+		Logger: logger.Default.LogMode(logger.Silent),
+	})
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	if err := db.Use(ftsgorm.Plugin()); err != nil {
+		log.Fatal(err)
+	}
+	if err := ftsgorm.Migrate(db, &Article{}); err != nil {
+		log.Fatal(err)
+	}
+
+	// Seed a few articles. The AFTER INSERT trigger maintains the
+	// FTS5 index automatically — no extra Go code.
+	articles := []Article{
+		{Title: "Hello world", Body: "The quick brown fox jumps over the lazy dog"},
+		{Title: "Bears are bears", Body: "Polar bears live in the arctic"},
+		{Title: "On dogs", Body: "Dogs are loyal companions"},
+	}
+	if err := db.Create(&articles).Error; err != nil {
+		log.Fatal(err)
+	}
+
+	// Search for "fox" — single hit with snippet + highlight pulled
+	// from the source via FTS5's documented auxiliary functions.
+	ctx := context.Background()
+	results, err := ftsgorm.Search[Article](
+		ctx, db, fts.Term("fox"),
+		ftsgorm.WithSnippet("body", "<b>", "</b>", "…", 8),
+		ftsgorm.WithHighlight("body", "[", "]"),
+	)
+	if err != nil {
+		log.Fatal(err)
+	}
+	fmt.Println("Search for 'fox':")
+	for _, r := range results {
+		fmt.Printf("  id=%d title=%q rank=%.4f\n", r.Model.ID, r.Model.Title, r.Rank)
+		fmt.Printf("    snippet:   %s\n", r.Snippet)
+		fmt.Printf("    highlight: %s\n", r.Highlight)
+	}
+
+	// Search across both columns with BM25 weights.
+	results, err = ftsgorm.Search[Article](
+		ctx, db, fts.Term("bears"),
+		ftsgorm.WithRanking(2.0, 1.0), // weight title higher than body
+	)
+	if err != nil {
+		log.Fatal(err)
+	}
+	fmt.Println("Search for 'bears' (title-weighted):")
+	for _, r := range results {
+		fmt.Printf("  id=%d title=%q\n", r.Model.ID, r.Model.Title)
+	}
+
+	// Cleanup: db.Migrator().DropTable also tears down the FTS5 table
+	// and its three triggers through the plugin's DropTableHook.
+	if err := db.Migrator().DropTable(&Article{}); err != nil {
+		log.Fatal(err)
+	}
+	fmt.Println("DropTable cascade: source + FTS5 table + triggers all gone")
+}

examples/gorm-vec-tagged/main.go

@@ -0,0 +1,80 @@
+// gorm-vec-tagged: the tag-driven flow for combining gorm models with
+// sqlite-vec. Mark an embedding field with `vec:"dim=N"` on a model,
+// register the vecgorm.Plugin(), call vecgorm.Migrate, and let
+// db.Create / vecgorm.KNN handle the sidecar transparently. No manual
+// vec.Table maintenance.
+//
+// Compare with examples/gorm-vec/ for the side-by-side recipe — both
+// patterns coexist; pick the one that fits.
+package main
+
+import (
+	"context"
+	"fmt"
+	"log"
+
+	"gorm.io/gorm"
+	"gorm.io/gorm/logger"
+
+	_ "github.com/go-again/sqlite"
+	sqlitegorm "github.com/go-again/sqlite/gorm"
+	vecgorm "github.com/go-again/sqlite/vec/gorm"
+)
+
+// Document is a typical gorm model with one tagged embedding field.
+// The vecgorm.Embedding wrapper type lets us skip the gorm:"-" tag —
+// the wrapper implements gorm's GormDataType interface so the schema
+// parser accepts it; the plugin then sets IgnoreMigration=true on the
+// field so no BLOB column lands on the source table.
+type Document struct {
+	ID        uint `gorm:"primaryKey"`
+	Title     string
+	Body      string
+	Embedding vecgorm.Embedding `vec:"dim=4;metric=cosine"`
+}
+
+func main() {
+	db, err := gorm.Open(sqlitegorm.Open(":memory:"), &gorm.Config{
+		Logger: logger.Default.LogMode(logger.Silent),
+	})
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	if err := db.Use(vecgorm.Plugin()); err != nil {
+		log.Fatal(err)
+	}
+	if err := vecgorm.Migrate(db, &Document{}); err != nil {
+		log.Fatal(err)
+	}
+
+	// Seed three documents. Embeddings populate the sidecar transparently.
+	docs := []Document{
+		{Title: "north", Body: "polar bears", Embedding: vecgorm.Embedding{0, 1, 0, 0}},
+		{Title: "east", Body: "sunrises", Embedding: vecgorm.Embedding{1, 0, 0, 0}},
+		{Title: "south", Body: "deserts", Embedding: vecgorm.Embedding{0, -1, 0, 0}},
+	}
+	if err := db.Create(&docs).Error; err != nil {
+		log.Fatal(err)
+	}
+
+	// Find the closest document to a query vector. KNN returns
+	// []Result[Document] with .Distance attached — no manual IN-clause
+	// or re-sorting required.
+	ctx := context.Background()
+	results, err := vecgorm.KNN[Document](ctx, db, []float32{0, 0.95, 0, 0}, 2)
+	if err != nil {
+		log.Fatal(err)
+	}
+	fmt.Println("KNN results:")
+	for _, r := range results {
+		fmt.Printf("  id=%d title=%q distance=%.4f\n", r.Model.ID, r.Model.Title, r.Distance)
+	}
+
+	// Cleanup is automatic via gorm's Migrator. db.Migrator().DropTable
+	// also drops the sidecar through the plugin's DropTableHook.
+	if err := db.Migrator().DropTable(&Document{}); err != nil {
+		log.Fatal(err)
+	}
+	fmt.Println("DropTable cascade: source + sidecar both gone")
+}

fts/fts.go

@@ -441,7 +441,7 @@ func (i *Index[K, V]) buildSearchSQL(q Query, cfg *searchConfig) (string, []any,
 	b.WriteString(" WHERE ")
 	b.WriteString(quote(i.name))
 	b.WriteString(" MATCH ?")
-	args = append(args, q.build())
+	args = append(args, q.Build())
 
 	if cfg.withRank {
 		b.WriteString(" ORDER BY __rank")