refactor(rebuild): inject cancellable runner

AGENTS.md

@@ -141,7 +141,7 @@ External knowledge sources are configured with active `*.inputs.toml` manifests
 - **V16 parser-backed message origin**: `search_items.origin` is constrained to `human|assistant|system|automation|unknown`, with an independent `origin_version`. Readers prove origin only from native structured envelopes and tool boundaries; stored text, historical `role`, Markdown headings, and frontmatter are never inference sources. Migration defaults historical rows to `unknown`; `PendingOriginPaths` reparses only available configured sources under the shared 200-path startup cap, so expired sources remain unknown. Stable identities may enrich `unknown` to a proven origin without replacing payload or degrading prior proof, and contradictory proof is rejected. Recovery preserves matching proof, enriches compatible unknown/proven duplicates, and rejects contradictory proven origins. `patterns --kind corrections --origin ...` is opt-in, filters before pagination, and emits origin only when requested; exact reader mappings and conditional output fields are owned by `docs/input-contract.md` and `docs/patterns.md`.
 - **F0a rich capture (migration v8)**: readers extract per-message identity and tool metadata BEFORE serialization/cleaning destroys the evidence — `uuid` (record uuid; tool blocks get stable `#tN`/`#rN` suffixes by block index), `tool_name`, `command_head`, `is_error` (`*bool`, three-valued: tool_result blocks carry it and it is paired back onto the tool_use message cross-record via `tool_use_id`), and `was_interrupted` (detected on raw content before `CleanContent` strips "Request interrupted"). Persisted to `search_items` (`extraction_version`, `was_interrupted` columns) and the perennial `tool_events` satellite table (`UNIQUE(source_path, ordinal)`, no CASCADE lifecycle — only `purge` deletes from it, explicitly). Claude reader only; Pi/OpenCode emit zero values and stay on the legacy path. Design: `docs/superpowers/specs/2026-07-17-pattern-discovery-northstar-design.md`.
 - **F0a.1 command-head extraction v2 (extraction_version bump to 2)**: The `commandHead()` function in `internal/readers/claude_reader.go` now strips leading POSIX variable assignments (tokens containing `=`) before extracting the actual command name. Example: `"SP=/path/to/code; go test ./..."` extracts `"go"` instead of `"SP=/path/to/code;"`. This reduces noise in sequence mining and command pattern discovery. The bump to `extraction_version=2` triggers B1 backfill on next sync, incrementally re-extracting command_head from stored text for files indexed before this change (up to 200 files/run).
-- **F0b perennial sync**: the DB is the perennial event store — session JSONL files expire (~30 days), indexed sessions survive them. Session files where EVERY message has a uuid sync append-only (no DELETE; `INSERT OR IGNORE` + UNIQUE constraints; row ids stable forever), with a one-time transition cleanup of legacy uuid-NULL rows per file. Files with any uuid-less message (Pi/OpenCode, legacy Claude) keep wipe-and-reload. `rebuild` is NON-destructive: after mutation-class startup sync has prepared the database, the handler re-derives both FTS indexes from `search_items` via FTS5 external-content `'rebuild'` in one transaction (`RebuildFTS()`) and performs no second sync — it never deletes rows and never re-reads disk as source of truth. `purge --before` is the only deletion path and deletes `tool_events` satellites explicitly in the same transaction (no CASCADE). The recover flow retains the remediation startup lock through post-install sync.
+- **F0b perennial sync**: the DB is the perennial event store — session JSONL files expire (~30 days), indexed sessions survive them. Session files where EVERY message has a uuid sync append-only (no DELETE; `INSERT OR IGNORE` + UNIQUE constraints; row ids stable forever), with a one-time transition cleanup of legacy uuid-NULL rows per file. Files with any uuid-less message (Pi/OpenCode, legacy Claude) keep wipe-and-reload. `rebuild` is NON-destructive. After mutation-class startup sync prepares the database, `RebuildFTSContext()` uses one transaction to clear and repopulate `messages_fts` only with `search_items.content_type IN ('text', 'code', 'reasoning')` and `tool_fts` only with `content_type = 'tool'`. It does not use the generic FTS5 external-content `'rebuild'` command because that command would cross-index external content. It does not modify `search_items` or re-read disk. `purge --before` is the only deletion path and deletes `tool_events` satellites explicitly in the same transaction (no CASCADE). The recover flow retains the remediation startup lock through post-install sync.
 - **F1 exit code mining**: the **reader** parses the exit code from the FULL tool output before `toolfmt` truncates it (`SerializeToolOutput` caps at 4000 runes and a Bash exit code is usually on the last line, so parsing the capped text lost exactly the codes worth having). `ExtractExitCodeText` does the parsing with no tool gate, because a `tool_result` block lives in a different JSONL record than its `tool_use` and the tool name is not knowable at that point; the Bash gate and the code are applied when result and use are paired by `tool_use_id` — the same cross-record pairing already used for `is_error`. `SyncFiles` persists `IndexedMessage.ExitCode` verbatim and MUST NOT re-derive a code from `msg.Text`, which is truncated. The extracted exit code is stored in the `tool_events.exit_code` column (migration v8; NULL for non-Bash tools or no match). The `patterns` command aggregates tool_events by (tool_name, command_head) for commands or (tool_name, is_error, exit_code) for failures, returning top N sorted by frequency with optional filters by project, session tag, and time window. Coverage metric reports the count of events with non-NULL is_error (signalled events) against the total failure count in the result set.
 - **F2 template mining (migration v10)**: unsupervised Drain-inspired template miner (`internal/templates/Miner`) discovers recurring error patterns from tool output during sync. Miner uses fixed-depth token prefix clustering (depth=2) to group messages; beyond the prefix, numeric/path/UUID tokens become `<*>` variables. Error-bearing lines (is_error=true) are extracted per tool via `ExtractErrorLines` (calibrated per tool: Bash prefers LAST non-empty line + error-matching; Go test matches "--- FAIL:", "FAIL\t", "error:"; others default to error-matching heuristic) and deterministically mined with SHA256 signature. Templates stored in `message_templates` (signature, normalization_version, template_text, occurrence_count, first_seen, last_seen) joined via `template_matches` (template_id, source_path, ordinal, item_uuid) with UNIQUE constraint for idempotency. Mining runs inside `SyncFiles` transaction; re-syncing increments occurrence_count only for new matches (detected via INSERT OR IGNORE). Query method `AggregateTemplates(opts)` filters by min_support (default 3), project, date range; `patterns --kind templates [--min-support N]` exposes results in text/JSON/robot formats with normalization_version metadata. _Q1 backfill update:_ Backfill mining filters rows by: include a tool-text row only when (it has case-insensitive "error: " prefix OR its ordinal has a tool_events row with is_error=1) AND it is not an input serialization.
 - **One line-selection predicate for both mining paths**: `shouldMineToolLine(contentType, text, isError)` in `internal/storage/mining.go` is called by sync-time `mineTemplatesForFile` and by backfill. Only the error signal differs — sync reads `IsError` off the message, backfill derives it from an `"error: "` prefix or a `tool_events` row. The predicate exists because the two paths had drifted, and the drift was the bug: sync selected on `ToolName != ""`, which picks the **tool_use** message, whose text _is_ the input serialization, while the error text lives on the **tool_result** message that carries no `ToolName`. Sync therefore mined inputs and never errors. Tool results have no tool name, so sync mines them as `"Unknown"`, matching backfill; per-tool line calibration is a follow-up.

cmd/backscroll/index_policy_test.go

@@ -550,14 +550,15 @@ func TestValidateRobotResultLinesRejectsNonResultLines(t *testing.T) {
 func TestRebuildFailsOnDerivedMaintenanceError(t *testing.T) {
 	dbPath := newSupportedIndexedConsumerDB(t)
 	setIndexPolicyEnv(t, dbPath, t.TempDir())
-	orig := rebuildBackfillDerived
-	rebuildBackfillDerived = func(*storage.Database, storage.BackfillDerivedOpts) error {
-		return fmt.Errorf("injected derived failure")
-	}
-	t.Cleanup(func() { rebuildBackfillDerived = orig })
+	runner := newRebuildRunner(&rebuildDependencies{
+		backfillDerivedContext: func(*storage.Database, context.Context, storage.BackfillDerivedOpts) error {
+			return fmt.Errorf("injected derived failure")
+		},
+	})
 
 	var stdout, stderr bytes.Buffer
-	err := run(&stdout, &stderr, []string{"rebuild"})
+	root := buildRootCmdWithDependencies(&stdout, &stderr, nil, nil, nil, runner)
+	err := runContextWithDependencies(context.Background(), &stdout, &stderr, []string{"rebuild"}, runDependencies{root: root})
 	if err == nil {
 		t.Fatalf("rebuild succeeded despite derived failure; stdout=%q stderr=%q", stdout.String(), stderr.String())
 	}

cmd/backscroll/rebuild.go

@@ -7,12 +7,81 @@ import (
 
 	"github.com/spf13/cobra"
 
+	"github.com/pablontiv/backscroll/internal/compat"
 	"github.com/pablontiv/backscroll/internal/config"
 	"github.com/pablontiv/backscroll/internal/projects"
 	"github.com/pablontiv/backscroll/internal/storage"
 )
 
-func newRebuildCmd(stdout, stderr io.Writer) *cobra.Command {
+type rebuildDependencies struct {
+	prepareIndex                         func(context.Context, *config.Config, indexCommandClass) (*storage.Database, *compat.Diagnostic, error)
+	closeIndexDB                         func(*storage.Database, error) error
+	rebuildFTSContext                    func(*storage.Database, context.Context) error
+	backfillDerivedContext               func(*storage.Database, context.Context, storage.BackfillDerivedOpts) error
+	reresolveProjectsContext             func(*storage.Database, context.Context, func(string) string) (int64, error)
+	loadRegistry                         func() projects.ProjectRegistry
+	reresolveProjectsWithRegistryContext func(*storage.Database, context.Context, projects.ProjectRegistry) (int64, error)
+}
+
+type rebuildRunner struct {
+	deps rebuildDependencies
+}
+
+func newRebuildRunner(deps *rebuildDependencies) *rebuildRunner {
+	resolved := rebuildDependencies{
+		prepareIndex: prepareIndex,
+		closeIndexDB: closeIndexDB,
+		rebuildFTSContext: func(db *storage.Database, ctx context.Context) error {
+			return db.RebuildFTSContext(ctx)
+		},
+		backfillDerivedContext: func(db *storage.Database, ctx context.Context, opts storage.BackfillDerivedOpts) error {
+			return db.BackfillDerivedContext(ctx, opts)
+		},
+		reresolveProjectsContext: func(db *storage.Database, ctx context.Context, resolver func(string) string) (int64, error) {
+			return db.ReresolveProjectsContext(ctx, resolver)
+		},
+		loadRegistry: projects.LoadGlobalRegistry,
+		reresolveProjectsWithRegistryContext: func(db *storage.Database, ctx context.Context, registry projects.ProjectRegistry) (int64, error) {
+			return db.ReresolveProjectsWithRegistryContext(ctx, registry)
+		},
+	}
+	if deps == nil {
+		return &rebuildRunner{deps: resolved}
+	}
+	if deps.prepareIndex != nil {
+		resolved.prepareIndex = deps.prepareIndex
+	}
+	if deps.closeIndexDB != nil {
+		resolved.closeIndexDB = deps.closeIndexDB
+	}
+	if deps.rebuildFTSContext != nil {
+		resolved.rebuildFTSContext = deps.rebuildFTSContext
+	}
+	if deps.backfillDerivedContext != nil {
+		resolved.backfillDerivedContext = deps.backfillDerivedContext
+	}
+	if deps.reresolveProjectsContext != nil {
+		resolved.reresolveProjectsContext = deps.reresolveProjectsContext
+	}
+	if deps.loadRegistry != nil {
+		resolved.loadRegistry = deps.loadRegistry
+	}
+	if deps.reresolveProjectsWithRegistryContext != nil {
+		resolved.reresolveProjectsWithRegistryContext = deps.reresolveProjectsWithRegistryContext
+	}
+	return &rebuildRunner{deps: resolved}
+}
+
+func newRebuildCmd(stdout, stderr io.Writer, runner *rebuildRunner) *cobra.Command {
+	if stdout == nil {
+		stdout = io.Discard
+	}
+	if stderr == nil {
+		stderr = io.Discard
+	}
+	if runner == nil {
+		runner = newRebuildRunner(nil)
+	}
 	cmd := &cobra.Command{
 		Use:          "rebuild",
 		Short:        "Rebuild the FTS search indexes from the database",
@@ -26,49 +95,55 @@ database is the perennial event store). Use 'purge' to delete data explicitly.`,
 			if startup.Config == nil {
 				return fmt.Errorf("startup configuration unavailable")
 			}
-			return runRebuild(cmd.Context(), stdout, stderr, startup.Config)
+			return runner.runRebuild(cmd.Context(), stdout, stderr, startup.Config)
 		},
 	}
 
 	return cmd
 }
 
-var (
-	rebuildBackfillDerived = func(db *storage.Database, opts storage.BackfillDerivedOpts) error {
-		return db.BackfillDerived(opts)
+func runRebuild(ctx context.Context, stdout, stderr io.Writer, cfg *config.Config) error {
+	return newRebuildRunner(nil).runRebuild(ctx, stdout, stderr, cfg)
+}
+
+func (r *rebuildRunner) runRebuild(ctx context.Context, stdout, stderr io.Writer, cfg *config.Config) (retErr error) {
+	if r == nil {
+		r = newRebuildRunner(nil)
 	}
-	rebuildReresolveProjects = func(db *storage.Database, ctx context.Context, resolver func(string) string) (int64, error) {
-		return db.ReresolveProjects(ctx, resolver)
+	if stdout == nil {
+		stdout = io.Discard
 	}
-	rebuildReresolveProjectsWithRegistry = func(db *storage.Database, ctx context.Context, registry projects.ProjectRegistry) (int64, error) {
-		return db.ReresolveProjectsWithRegistry(ctx, registry)
+	if stderr == nil {
+		stderr = io.Discard
 	}
-)
 
-func runRebuild(ctx context.Context, stdout, stderr io.Writer, cfg *config.Config) (retErr error) {
-	db, diag, err := prepareIndex(ctx, cfg, indexMutation)
+	db, diag, err := r.deps.prepareIndex(ctx, cfg, indexMutation)
 	if diag != nil {
 		return refuseIndex(stdout, stderr, *diag, false, false)
 	}
 	if err != nil {
 		return fmt.Errorf("prepare index: %w", err)
 	}
-	defer func() { retErr = closeIndexDB(db, retErr) }()
+	defer func() { retErr = r.deps.closeIndexDB(db, retErr) }()
 
-	_, _ = fmt.Fprintf(stdout, "Re-deriving FTS indexes from database...\n")
-	if err := db.RebuildFTS(); err != nil {
+	if err := writeRebuildOutput(stdout, "Re-deriving FTS indexes from database...\n"); err != nil {
+		return err
+	}
+	if err := r.deps.rebuildFTSContext(db, ctx); err != nil {
 		return fmt.Errorf("rebuild FTS: %w", err)
 	}
 
-	// Backfill templates, corrections, and lossy tool_events for expired files
-	_, _ = fmt.Fprintf(stdout, "Backfilling derived data from stored text...\n")
+	// Backfill templates, corrections, and lossy tool_events for expired files.
+	if err := writeRebuildOutput(stdout, "Backfilling derived data from stored text...\n"); err != nil {
+		return err
+	}
 	var backfillStats struct {
 		filesProcessed  int
 		templatesFound  int
 		signalsFound    int
 		eventsExtracted int
 	}
-	if err := rebuildBackfillDerived(db, storage.BackfillDerivedOpts{
+	if err := r.deps.backfillDerivedContext(db, ctx, storage.BackfillDerivedOpts{
 		OnProgress: func(processed, templateCount, signalCount, eventCount int) {
 			backfillStats.filesProcessed = processed
 			backfillStats.templatesFound = templateCount
@@ -79,36 +154,52 @@ func runRebuild(ctx context.Context, stdout, stderr io.Writer, cfg *config.Confi
 		return fmt.Errorf("backfill derived: %w", err)
 	}
 	if backfillStats.filesProcessed > 0 {
-		_, _ = fmt.Fprintf(stdout, "Backfill complete: %d files processed, %d templates, %d corrections, %d lossy events.\n",
-			backfillStats.filesProcessed, backfillStats.templatesFound, backfillStats.signalsFound, backfillStats.eventsExtracted)
+		if err := writeRebuildOutput(stdout, "Backfill complete: %d files processed, %d templates, %d corrections, %d lossy events.\n",
+			backfillStats.filesProcessed, backfillStats.templatesFound, backfillStats.signalsFound, backfillStats.eventsExtracted); err != nil {
+			return err
+		}
 	}
 
-	// Re-resolve project identities from session paths
-	_, _ = fmt.Fprintf(stdout, "Re-resolving projects from session paths...\n")
+	// Re-resolve project identities from session paths.
+	if err := writeRebuildOutput(stdout, "Re-resolving projects from session paths...\n"); err != nil {
+		return err
+	}
 	resolver := func(sourcePath string) string {
 		cwd := projects.DecodeCwdFromSessionPath(sourcePath)
 		if cwd != "" {
 			return projects.DeriveFallbackID(cwd)
 		}
 		return ""
 	}
-	resolved, err := rebuildReresolveProjects(db, ctx, resolver)
+	resolved, err := r.deps.reresolveProjectsContext(db, ctx, resolver)
 	if err != nil {
 		return fmt.Errorf("project re-resolution: %w", err)
 	} else if resolved > 0 {
-		_, _ = fmt.Fprintf(stdout, "Re-resolved %d sessions with derived project identities.\n", resolved)
+		if err := writeRebuildOutput(stdout, "Re-resolved %d sessions with derived project identities.\n", resolved); err != nil {
+			return err
+		}
 	}
 
-	// Registry-aware re-resolution: correct historical fallback labels
-	_, _ = fmt.Fprintf(stdout, "Checking registry for project label corrections...\n")
-	registry := projects.LoadGlobalRegistry()
-	registryMatched, err := rebuildReresolveProjectsWithRegistry(db, ctx, registry)
+	// Registry-aware re-resolution corrects historical fallback labels.
+	if err := writeRebuildOutput(stdout, "Checking registry for project label corrections...\n"); err != nil {
+		return err
+	}
+	registry := r.deps.loadRegistry()
+	registryMatched, err := r.deps.reresolveProjectsWithRegistryContext(db, ctx, registry)
 	if err != nil {
 		return fmt.Errorf("registry re-resolution: %w", err)
 	} else if registryMatched > 0 {
-		_, _ = fmt.Fprintf(stdout, "Registry matched and corrected %d sessions.\n", registryMatched)
+		if err := writeRebuildOutput(stdout, "Registry matched and corrected %d sessions.\n", registryMatched); err != nil {
+			return err
+		}
 	}
 
-	_, _ = fmt.Fprintf(stdout, "Rebuild complete. No indexed data was deleted (perennity contract).\n")
+	return writeRebuildOutput(stdout, "Rebuild complete. No indexed data was deleted (perennity contract).\n")
+}
+
+func writeRebuildOutput(w io.Writer, format string, args ...any) error {
+	if _, err := fmt.Fprintf(w, format, args...); err != nil {
+		return fmt.Errorf("write rebuild output: %w", err)
+	}
 	return nil
 }