Docs: the in-process probe, batched scan writes, added_at from discovery

docs-developers/contributing/gates-and-ci.md

@@ -49,6 +49,7 @@ The workflows for the server, player and metadata repos, verified against
 | `ci.yml` | `ci` | every PR; push to `main` | Three jobs side by side. **test**: Go from `go.mod`, Node from `admin-ui/.nvmrc` and `scripts/build-admin.sh --vite-only` (before the Go steps, so the embed tests see a real build; no typecheck, the admin-ui job does that), installs ffmpeg (so the ffprobe-dependent scanner tests stay live), `go test -race -coverprofile=coverage.out ./...` (which compiles every package, test files or not, so there is no separate build step), uploads the coverage artifact. **admin-ui**: the full `scripts/build-admin.sh` (`npm ci`, `npm run check`, the build). **lint**: `golangci-lint-action@v9` (golangci-lint v2, config `.golangci.yml`; its govet replaces a separate `go vet` step). |
 | `image.yml` | `server image` | `v*` tags; push to `main`; manual dispatch (input `web_version`) | Builds the Docker image, baking the pinned web player in via the `WEB_IMAGE` build-arg, and pushes `ghcr.io/<owner>/audiosilo-server` (semver + `latest` on tags; `edge` on `main`; `sha-<short>` off a tag). See [releasing](releasing.md#testing-main-before-a-release-edge). |
 | `release.yml` | `release (native binaries)` | `v*` tags; manual dispatch (input `web_version`) | GoReleaser: cross-platform native binaries with the web player embedded (`-tags embedplayer`), published as a **draft** GitHub Release. |
+| `probe-drift.yml` | `probe-drift` | PRs and pushes to `main` touching `internal/metadata/**`, the `Dockerfile` or the workflow; weekly (Monday 05:17 UTC); manual dispatch | Job **image-ffprobe**: builds the `internal/metadata` test binary, then runs it in the Dockerfile's Alpine base image with that image's ffmpeg installed and `AUDIOSILO_PROBE_LIVE=1`: `TestProbeMatchesLiveFFprobe` and `TestProbeMatchesGoldens`. It fails when the [in-process probe](../server/scanner.md#the-in-process-probe) answers a fixture differently from the image's ffprobe (drift), or when that ffprobe is missing or of a release the probe has no rules for. Weekly because Alpine updates ffmpeg with no change in the repo. See [The in-process probe's fixtures](#the-in-process-probes-fixtures). |
 
 ### audiosilo-frontend
 
@@ -123,6 +124,44 @@ the repo:
   (`handlers_members.go`), the rate limiters, `auth.ResolveRequest`/`lookupToken`, `web.htmlCSP`, or `bearerToken`. See
   [auth and security](../server/auth-and-security.md).
 
+### The in-process probe's fixtures
+
+The scanner's [in-process probe](../server/scanner.md#the-in-process-probe)
+must answer exactly as ffprobe does, so it is tested against ffprobe's own
+answers, with and without ffprobe installed:
+
+- **The corpus.** `internal/metadata/testdata/probe` holds about 90 small
+  fixtures (a few seconds of audio each, under 3 MB together) in every shape
+  the probe reads or leaves to ffprobe: MP3 (CBR, VBR with and without a
+  Xing/VBRI header, gapless, ID3 and APE tags, chapters), M4B and M4A, FLAC,
+  Ogg Vorbis/Opus/FLAC (`.ogg` and `.opus`) and MKA, plus crafted and truncated files.
+  `scripts/gen-probe-fixtures.sh` makes them (with `cmd/mkfixtures` for the
+  crafted ones; it needs ffmpeg/ffprobe 7.x or 8.x with libmp3lame, libvorbis
+  and libopus, the `lame` CLI and `metaflac`). Regenerating rewrites every
+  fixture, since encoders differ by version: commit the files and the goldens
+  together and review the goldens' diff.
+- **Goldens.** `golden-<release>.json` (`golden-7.1.json`, `golden-8.1.json`)
+  is a known release's normalised answer for every fixture, written by
+  `cmd/probediff -golden`; the script writes one for the local ffprobe and,
+  through docker, one for the Docker image's. `probe_golden_test.go` checks the
+  probe against each with no ffprobe needed, so the plain `go test ./...` gate
+  covers it everywhere.
+- **The live comparison.** `probe_live_test.go` compares the probe with the
+  ffprobe on `PATH` over the fixtures, and over any folders of real books
+  `AUDIOSILO_PROBE_DIRS` (a path list) names. It skips without an ffprobe or
+  with one of a release the probe has no rules for, unless
+  `AUDIOSILO_PROBE_LIVE` is set, when either is a failure (the `probe-drift`
+  workflow sets it).
+- **Adding a release.** Its rules where they differ, its golden and a
+  `knownRules` entry; then the live test and the CI job cover it.
+- **Stand-in ffprobe scripts.** A test whose ffprobe is a script on *valid*
+  files (to fake a rejection or a time-out) would otherwise be answered by the
+  probe and never run the script. It calls
+  `metadata.ForceFFprobeForTests(path)` (`library`'s `stubProbe` helper does),
+  which sends every `Extract` given that ffprobe path to ffprobe; it is keyed
+  by the path, so parallel tests don't touch each other, and panics outside a
+  test binary.
+
 ### Frontend (TypeScript)
 
 - Harness: **jest-expo (jest 29) + `@testing-library/react-native` 14** -
@@ -141,7 +180,9 @@ the repo:
 - **ffmpeg-dependent tests skip silently.** A few server scanner tests need
   ffprobe; without it they `t.Skip` and the suite still passes - so a green local
   run without ffmpeg proved less than you think. CI installs ffmpeg precisely to
-  keep those tests live. Install it locally too.
+  keep those tests live. Install it locally too. (The in-process probe's golden
+  test needs no ffprobe; its live comparison skips without one, see
+  [above](#the-in-process-probes-fixtures).)
 - **Keep `package-lock.json` committed in sync.** CI uses `npm ci` (frozen
   lockfile), so after any dependency change regenerate the lockfile with
   `npm install` and commit it - otherwise CI fails on a lockfile mismatch.

docs-developers/contributing/workspace.md

@@ -54,7 +54,7 @@ cd ~/dev/audiosilo && scripts/bootstrap.sh
 | **Go 1.26+** | server, audiosilo-meta, the Bridge | Each `go.mod` sets the minimum `go` version (the server's also pins a `toolchain`); with the default `GOTOOLCHAIN=auto`, any Go 1.21+ fetches the toolchain it needs on first use. |
 | **Node 24** | frontend, the server's admin console, this docs site | `.nvmrc` pins `24.16.0` in `audiosilo-frontend` and in `audiosilo-server/admin-ui` - run `nvm use` in each. **Older Node breaks Expo**: React Native 0.85 needs ≥ 20.19.4, and the Expo CLI's env-file loader uses `util.parseEnv` (Node ≥ 20.12), so an old system `node` crashes the moment a `.env` file exists. |
 | **golangci-lint v2** | server and audiosilo-meta lint gates | v2 has been required since Go 1.25. Config is each repo's `.golangci.yml`, adopted at a *green baseline* (see [gates and CI](gates-and-ci.md)). |
-| **ffmpeg / ffprobe** | server scanner tests, transcoding | Optional at runtime (the server degrades gracefully), but a few scanner tests `t.Skip` without ffprobe - install it so they actually run (CI installs it). |
+| **ffmpeg / ffprobe** | server scanner tests, transcoding | Optional at runtime (the server degrades gracefully), but a few scanner tests `t.Skip` without ffprobe - install it so they actually run (CI installs it). Use 7.1 or 8.1 to run the in-process probe's live comparison; regenerating its fixtures also needs `lame` and `metaflac` (see [gates and CI](gates-and-ci.md#the-in-process-probes-fixtures)). |
 | **Docker** | container images, `scripts/fetch-web-player.sh` | Only needed for image builds / release work, not day-to-day dev. |
 | **Playwright** | regenerating documentation screenshots | Only if you work on the docs screenshot pipeline (`audiosilo-docs/screenshots/`) or the marketing/store asset runbook (workspace `SCREENSHOTS.md`). See [documentation](documentation.md). |
 | **Xcode / Android Studio (JDK 17 + SDK)** | native frontend dev builds | Only for running the player on iOS/Android; web dev needs neither. |

docs-developers/server/community-chapters.md

@@ -72,7 +72,7 @@ admin asks for always fits again (a first fit made without ffmpeg, say, gets
 its pauses).
 
 A `Kick` follows every committed book change: the API registers it with
-`Catalog.OnBookChange`, which `UpsertBook` (a scan), `EditBook` and `EditBooks`
+`Catalog.OnBookChange`, which `UpsertBooks` and `UpsertBook` (a scan), `EditBook` and `EditBooks`
 (an edit, an accepted match, a [bulk match run](api/reference.md#bulk-community-matching)
 applying), `SetEnrichment`, clearing community matches and setting or removing a
 custom cover fire after their transaction (to every listener: the

docs-developers/server/configuration.md

@@ -52,7 +52,7 @@ updates only a declared library's root and default view).
 | Flag | Default | Meaning |
 |---|---|---|
 | `--data` | `./data` | Data directory: config, SQLite database, generated certs, downloaded tools, the Matroska remux cache (`cache/remux/`) |
-| `--ffprobe` | `ffprobe` | Path to ffprobe (durations, chapters, codec detection). **`""` disables it** - the scanner degrades to path-derived metadata and `direct_playable` defaults to true |
+| `--ffprobe` | `ffprobe` | Path to ffprobe (durations, chapters, codec detection). With FFmpeg 7.1 or 8.1 the scanner reads most files in-process by that release's rules and runs it only for the rest; with any other release ffprobe reads every file (see [The in-process probe](scanner.md#the-in-process-probe)). **`""` disables it**, the in-process probe included - the scanner degrades to path-derived metadata and `direct_playable` defaults to true |
 | `--ffmpeg` | `ffmpeg` | Path to ffmpeg for on-the-fly transcoding and the MP3-in-Matroska remux. **`""` disables it** - `?transcode=1` returns 503, the `transcode` capability reports false, and an `.mka` is served as stored (no remux, `direct_playable: false`) |
 | `--setup` | `false` | First-run **setup page**: instead of auto-creating the admin and printing credentials once, mint a one-time token and enable the guarded `/setup` page, where the admin's username and password are chosen in the browser (see [Built-in web UI](web-ui.md#first-run-setup-page-setup)). Either way the admin console's [first-run setup flow](web-ui.md#the-first-run-setup-flow) then adds the books folder and the rest on the first `/admin` visit |
 
@@ -276,14 +276,16 @@ feed/audio network limits, allowlists and the per-episode size cap in its
 
 | Key | Type / default | Meaning |
 |---|---|---|
-| `scan.concurrency` | int, `0` | How many audio files a scan reads at once (their tags, then ffprobe), `0` to `64`. `0` means one per CPU, up to 8 (`library.DefaultScanReaders`). CPUs are counted by `GOMAXPROCS`, so in a container that is its CPU limit, not the host's cores. Discovery lists folders and looks files up 8 at once, or fewer when this is set below 8 (see [A rescan that finds nothing changed](scanner.md#a-rescan-that-finds-nothing-changed)). Lower it for a library on a slow disk or a busy share; raise it for a fast disk on a machine with cores to spare. Read once at start (`Scanner.SetConcurrency`), with **no console setting**; `AUDIOSILO_SCAN_CONCURRENCY` sets it too |
+| `scan.concurrency` | int, `0` | How many audio files a scan reads at once (their tags and the in-process probe, then ffprobe for a file the probe leaves to it), `0` to `64`. `0` means one per CPU, up to 8 (`library.DefaultScanReaders`). CPUs are counted by `GOMAXPROCS`, so in a container that is its CPU limit, not the host's cores. Discovery lists folders and looks files up 8 at once, or fewer when this is set below 8 (see [A rescan that finds nothing changed](scanner.md#a-rescan-that-finds-nothing-changed)). Lower it for a library on a slow disk or a busy share; raise it for a fast disk on a machine with cores to spare. Read once at start (`Scanner.SetConcurrency`), with **no console setting**; `AUDIOSILO_SCAN_CONCURRENCY` sets it too |
 
 The index comes out the same however many files a scan reads at once (see
 [Reading several files at once](scanner.md#reading-several-files-at-once)).
 The cap of 8 for the default is deliberate: past about eight, each ffprobe
 run costs more CPU as they contend. On a 20-core machine, 20 at once finished
 the same scan sooner than 8 (8.0 s against 13.4 s) but took about a third more
-CPU (141 against 104 CPU-seconds).
+CPU (141 against 104 CPU-seconds). Those figures are for ffprobe on every file;
+with the in-process probe, a first scan spends far less time per file
+reading, and the single writer and the tag reads are most of what is left.
 
 ```yaml
 scan:
@@ -931,6 +933,13 @@ tools):
   Rosetta 2 on Apple Silicon, which is fine to exec from an arm64 server).
 - **Self-check**: every downloaded binary must successfully run `-version`
   (15 s timeout) before it is adopted; a failing binary is deleted.
+- **No in-process probe with the Linux/Windows download**: BtbN's builds are
+  FFmpeg master (`ffmpeg-master-latest-*`), whose version is no release, so the
+  scanner's [in-process probe](scanner.md#the-in-process-probe) stays off and
+  ffprobe reads every file. evermeet's macOS download is the latest release
+  when it was fetched (and is kept), so the probe runs with it only while that
+  is 7.1 or 8.1. A local ffprobe of a known release (7.1 or 8.1) found first
+  avoids both.
 - **Hardening**: extraction only takes files named `ffmpeg`/`ffprobe`
   (basename-only, so no zip-slip), capped at 300 MiB per binary
   (decompression-bomb guard). Pinning per-asset SHA-256 is noted in the

docs-developers/server/media.md

@@ -313,7 +313,8 @@ first full pass the runner only reads books whose art is new.
 
 ## `DirectPlayable`: when a client should transcode
 
-The scanner records each book's audio codec (ffprobe `codec_name`, verbatim)
+The scanner records each book's audio codec (ffprobe `codec_name`, verbatim,
+or the [in-process probe](scanner.md#the-in-process-probe)'s identical answer)
 in `books.codec`. `media.DirectPlayable(format, codec)` reports whether mainstream
 browsers can play the **stored** file natively. A **container** no browser plays
 (`containerFormats`: `mka`, by the file extension in `books.format`) never is,

docs-developers/server/overview.md

@@ -162,7 +162,10 @@ once while the index is written in order). See [Scanner](scanner.md).
 ### `internal/metadata`
 
 Metadata extraction: embedded tags in-process via `dhowden/tag`, durations /
-chapters / codec via ffprobe when available (`probe.go`), the release
+chapters / codec when ffprobe is available, read in-process for MP3, MP4, FLAC,
+Ogg and MKA by the configured ffprobe's release's rules (`fastprobe*.go`, the
+[in-process probe](scanner.md#the-in-process-probe)) and by ffprobe itself for
+any file the probe is unsure of (`probe.go`), the release
 date (`ReleaseDate`, a date tag as `YYYY[-MM[-DD]]`), and
 `DeriveFromPath` - the structural path heuristic
 (`Author/Series/01 - Title.m4b`; a lone folder above the book is its author,
@@ -428,6 +431,9 @@ Every feature ships with a test (see
   `newTestCatalog`), `internal/web/web_test.go`, and so on.
 - A few scanner tests need `ffprobe` on the machine; without it they `t.Skip`
   (CI installs ffmpeg).
+- The in-process probe is checked against ffprobe's answers over a fixture
+  corpus; see
+  [Gates and CI](../contributing/gates-and-ci.md#the-in-process-probes-fixtures).
 - **Security-critical code requires both an allowed and a denied regression
   test** - the enumerated list is in
   [Auth & security](auth-and-security.md#the-allowed--denied-test-rule).