diff --git a/docs-developers/contributing/gates-and-ci.md b/docs-developers/contributing/gates-and-ci.md index d2f9d892..76f5e668 100644 --- a/docs-developers/contributing/gates-and-ci.md +++ b/docs-developers/contributing/gates-and-ci.md @@ -49,6 +49,7 @@ The workflows for the server, player and metadata repos, verified against | `ci.yml` | `ci` | every PR; push to `main` | Three jobs side by side. **test**: Go from `go.mod`, Node from `admin-ui/.nvmrc` and `scripts/build-admin.sh --vite-only` (before the Go steps, so the embed tests see a real build; no typecheck, the admin-ui job does that), installs ffmpeg (so the ffprobe-dependent scanner tests stay live), `go test -race -coverprofile=coverage.out ./...` (which compiles every package, test files or not, so there is no separate build step), uploads the coverage artifact. **admin-ui**: the full `scripts/build-admin.sh` (`npm ci`, `npm run check`, the build). **lint**: `golangci-lint-action@v9` (golangci-lint v2, config `.golangci.yml`; its govet replaces a separate `go vet` step). | | `image.yml` | `server image` | `v*` tags; push to `main`; manual dispatch (input `web_version`) | Builds the Docker image, baking the pinned web player in via the `WEB_IMAGE` build-arg, and pushes `ghcr.io//audiosilo-server` (semver + `latest` on tags; `edge` on `main`; `sha-` off a tag). See [releasing](releasing.md#testing-main-before-a-release-edge). | | `release.yml` | `release (native binaries)` | `v*` tags; manual dispatch (input `web_version`) | GoReleaser: cross-platform native binaries with the web player embedded (`-tags embedplayer`), published as a **draft** GitHub Release. | +| `probe-drift.yml` | `probe-drift` | PRs and pushes to `main` touching `internal/metadata/**`, the `Dockerfile` or the workflow; weekly (Monday 05:17 UTC); manual dispatch | Job **image-ffprobe**: builds the `internal/metadata` test binary, then runs it in the Dockerfile's Alpine base image with that image's ffmpeg installed and `AUDIOSILO_PROBE_LIVE=1`: `TestProbeMatchesLiveFFprobe` and `TestProbeMatchesGoldens`. It fails when the [in-process probe](../server/scanner.md#the-in-process-probe) answers a fixture differently from the image's ffprobe (drift), or when that ffprobe is missing or of a release the probe has no rules for. Weekly because Alpine updates ffmpeg with no change in the repo. See [The in-process probe's fixtures](#the-in-process-probes-fixtures). | ### audiosilo-frontend @@ -123,6 +124,44 @@ the repo: (`handlers_members.go`), the rate limiters, `auth.ResolveRequest`/`lookupToken`, `web.htmlCSP`, or `bearerToken`. See [auth and security](../server/auth-and-security.md). +### The in-process probe's fixtures + +The scanner's [in-process probe](../server/scanner.md#the-in-process-probe) +must answer exactly as ffprobe does, so it is tested against ffprobe's own +answers, with and without ffprobe installed: + +- **The corpus.** `internal/metadata/testdata/probe` holds about 90 small + fixtures (a few seconds of audio each, under 3 MB together) in every shape + the probe reads or leaves to ffprobe: MP3 (CBR, VBR with and without a + Xing/VBRI header, gapless, ID3 and APE tags, chapters), M4B and M4A, FLAC, + Ogg Vorbis/Opus/FLAC (`.ogg` and `.opus`) and MKA, plus crafted and truncated files. + `scripts/gen-probe-fixtures.sh` makes them (with `cmd/mkfixtures` for the + crafted ones; it needs ffmpeg/ffprobe 7.x or 8.x with libmp3lame, libvorbis + and libopus, the `lame` CLI and `metaflac`). Regenerating rewrites every + fixture, since encoders differ by version: commit the files and the goldens + together and review the goldens' diff. +- **Goldens.** `golden-.json` (`golden-7.1.json`, `golden-8.1.json`) + is a known release's normalised answer for every fixture, written by + `cmd/probediff -golden`; the script writes one for the local ffprobe and, + through docker, one for the Docker image's. `probe_golden_test.go` checks the + probe against each with no ffprobe needed, so the plain `go test ./...` gate + covers it everywhere. +- **The live comparison.** `probe_live_test.go` compares the probe with the + ffprobe on `PATH` over the fixtures, and over any folders of real books + `AUDIOSILO_PROBE_DIRS` (a path list) names. It skips without an ffprobe or + with one of a release the probe has no rules for, unless + `AUDIOSILO_PROBE_LIVE` is set, when either is a failure (the `probe-drift` + workflow sets it). +- **Adding a release.** Its rules where they differ, its golden and a + `knownRules` entry; then the live test and the CI job cover it. +- **Stand-in ffprobe scripts.** A test whose ffprobe is a script on *valid* + files (to fake a rejection or a time-out) would otherwise be answered by the + probe and never run the script. It calls + `metadata.ForceFFprobeForTests(path)` (`library`'s `stubProbe` helper does), + which sends every `Extract` given that ffprobe path to ffprobe; it is keyed + by the path, so parallel tests don't touch each other, and panics outside a + test binary. + ### Frontend (TypeScript) - Harness: **jest-expo (jest 29) + `@testing-library/react-native` 14** - @@ -141,7 +180,9 @@ the repo: - **ffmpeg-dependent tests skip silently.** A few server scanner tests need ffprobe; without it they `t.Skip` and the suite still passes - so a green local run without ffmpeg proved less than you think. CI installs ffmpeg precisely to - keep those tests live. Install it locally too. + keep those tests live. Install it locally too. (The in-process probe's golden + test needs no ffprobe; its live comparison skips without one, see + [above](#the-in-process-probes-fixtures).) - **Keep `package-lock.json` committed in sync.** CI uses `npm ci` (frozen lockfile), so after any dependency change regenerate the lockfile with `npm install` and commit it - otherwise CI fails on a lockfile mismatch. diff --git a/docs-developers/contributing/workspace.md b/docs-developers/contributing/workspace.md index f0d3a078..be35d939 100644 --- a/docs-developers/contributing/workspace.md +++ b/docs-developers/contributing/workspace.md @@ -54,7 +54,7 @@ cd ~/dev/audiosilo && scripts/bootstrap.sh | **Go 1.26+** | server, audiosilo-meta, the Bridge | Each `go.mod` sets the minimum `go` version (the server's also pins a `toolchain`); with the default `GOTOOLCHAIN=auto`, any Go 1.21+ fetches the toolchain it needs on first use. | | **Node 24** | frontend, the server's admin console, this docs site | `.nvmrc` pins `24.16.0` in `audiosilo-frontend` and in `audiosilo-server/admin-ui` - run `nvm use` in each. **Older Node breaks Expo**: React Native 0.85 needs ≥ 20.19.4, and the Expo CLI's env-file loader uses `util.parseEnv` (Node ≥ 20.12), so an old system `node` crashes the moment a `.env` file exists. | | **golangci-lint v2** | server and audiosilo-meta lint gates | v2 has been required since Go 1.25. Config is each repo's `.golangci.yml`, adopted at a *green baseline* (see [gates and CI](gates-and-ci.md)). | -| **ffmpeg / ffprobe** | server scanner tests, transcoding | Optional at runtime (the server degrades gracefully), but a few scanner tests `t.Skip` without ffprobe - install it so they actually run (CI installs it). | +| **ffmpeg / ffprobe** | server scanner tests, transcoding | Optional at runtime (the server degrades gracefully), but a few scanner tests `t.Skip` without ffprobe - install it so they actually run (CI installs it). Use 7.1 or 8.1 to run the in-process probe's live comparison; regenerating its fixtures also needs `lame` and `metaflac` (see [gates and CI](gates-and-ci.md#the-in-process-probes-fixtures)). | | **Docker** | container images, `scripts/fetch-web-player.sh` | Only needed for image builds / release work, not day-to-day dev. | | **Playwright** | regenerating documentation screenshots | Only if you work on the docs screenshot pipeline (`audiosilo-docs/screenshots/`) or the marketing/store asset runbook (workspace `SCREENSHOTS.md`). See [documentation](documentation.md). | | **Xcode / Android Studio (JDK 17 + SDK)** | native frontend dev builds | Only for running the player on iOS/Android; web dev needs neither. | diff --git a/docs-developers/server/community-chapters.md b/docs-developers/server/community-chapters.md index 8783e3d4..f1dd81ee 100644 --- a/docs-developers/server/community-chapters.md +++ b/docs-developers/server/community-chapters.md @@ -72,7 +72,7 @@ admin asks for always fits again (a first fit made without ffmpeg, say, gets its pauses). A `Kick` follows every committed book change: the API registers it with -`Catalog.OnBookChange`, which `UpsertBook` (a scan), `EditBook` and `EditBooks` +`Catalog.OnBookChange`, which `UpsertBooks` and `UpsertBook` (a scan), `EditBook` and `EditBooks` (an edit, an accepted match, a [bulk match run](api/reference.md#bulk-community-matching) applying), `SetEnrichment`, clearing community matches and setting or removing a custom cover fire after their transaction (to every listener: the diff --git a/docs-developers/server/configuration.md b/docs-developers/server/configuration.md index 2a6e6aa1..7a2e5c72 100644 --- a/docs-developers/server/configuration.md +++ b/docs-developers/server/configuration.md @@ -52,7 +52,7 @@ updates only a declared library's root and default view). | Flag | Default | Meaning | |---|---|---| | `--data` | `./data` | Data directory: config, SQLite database, generated certs, downloaded tools, the Matroska remux cache (`cache/remux/`) | -| `--ffprobe` | `ffprobe` | Path to ffprobe (durations, chapters, codec detection). **`""` disables it** - the scanner degrades to path-derived metadata and `direct_playable` defaults to true | +| `--ffprobe` | `ffprobe` | Path to ffprobe (durations, chapters, codec detection). With FFmpeg 7.1 or 8.1 the scanner reads most files in-process by that release's rules and runs it only for the rest; with any other release ffprobe reads every file (see [The in-process probe](scanner.md#the-in-process-probe)). **`""` disables it**, the in-process probe included - the scanner degrades to path-derived metadata and `direct_playable` defaults to true | | `--ffmpeg` | `ffmpeg` | Path to ffmpeg for on-the-fly transcoding and the MP3-in-Matroska remux. **`""` disables it** - `?transcode=1` returns 503, the `transcode` capability reports false, and an `.mka` is served as stored (no remux, `direct_playable: false`) | | `--setup` | `false` | First-run **setup page**: instead of auto-creating the admin and printing credentials once, mint a one-time token and enable the guarded `/setup` page, where the admin's username and password are chosen in the browser (see [Built-in web UI](web-ui.md#first-run-setup-page-setup)). Either way the admin console's [first-run setup flow](web-ui.md#the-first-run-setup-flow) then adds the books folder and the rest on the first `/admin` visit | @@ -276,14 +276,16 @@ feed/audio network limits, allowlists and the per-episode size cap in its | Key | Type / default | Meaning | |---|---|---| -| `scan.concurrency` | int, `0` | How many audio files a scan reads at once (their tags, then ffprobe), `0` to `64`. `0` means one per CPU, up to 8 (`library.DefaultScanReaders`). CPUs are counted by `GOMAXPROCS`, so in a container that is its CPU limit, not the host's cores. Discovery lists folders and looks files up 8 at once, or fewer when this is set below 8 (see [A rescan that finds nothing changed](scanner.md#a-rescan-that-finds-nothing-changed)). Lower it for a library on a slow disk or a busy share; raise it for a fast disk on a machine with cores to spare. Read once at start (`Scanner.SetConcurrency`), with **no console setting**; `AUDIOSILO_SCAN_CONCURRENCY` sets it too | +| `scan.concurrency` | int, `0` | How many audio files a scan reads at once (their tags and the in-process probe, then ffprobe for a file the probe leaves to it), `0` to `64`. `0` means one per CPU, up to 8 (`library.DefaultScanReaders`). CPUs are counted by `GOMAXPROCS`, so in a container that is its CPU limit, not the host's cores. Discovery lists folders and looks files up 8 at once, or fewer when this is set below 8 (see [A rescan that finds nothing changed](scanner.md#a-rescan-that-finds-nothing-changed)). Lower it for a library on a slow disk or a busy share; raise it for a fast disk on a machine with cores to spare. Read once at start (`Scanner.SetConcurrency`), with **no console setting**; `AUDIOSILO_SCAN_CONCURRENCY` sets it too | The index comes out the same however many files a scan reads at once (see [Reading several files at once](scanner.md#reading-several-files-at-once)). The cap of 8 for the default is deliberate: past about eight, each ffprobe run costs more CPU as they contend. On a 20-core machine, 20 at once finished the same scan sooner than 8 (8.0 s against 13.4 s) but took about a third more -CPU (141 against 104 CPU-seconds). +CPU (141 against 104 CPU-seconds). Those figures are for ffprobe on every file; +with the in-process probe, a first scan spends far less time per file +reading, and the single writer and the tag reads are most of what is left. ```yaml scan: @@ -931,6 +933,13 @@ tools): Rosetta 2 on Apple Silicon, which is fine to exec from an arm64 server). - **Self-check**: every downloaded binary must successfully run `-version` (15 s timeout) before it is adopted; a failing binary is deleted. +- **No in-process probe with the Linux/Windows download**: BtbN's builds are + FFmpeg master (`ffmpeg-master-latest-*`), whose version is no release, so the + scanner's [in-process probe](scanner.md#the-in-process-probe) stays off and + ffprobe reads every file. evermeet's macOS download is the latest release + when it was fetched (and is kept), so the probe runs with it only while that + is 7.1 or 8.1. A local ffprobe of a known release (7.1 or 8.1) found first + avoids both. - **Hardening**: extraction only takes files named `ffmpeg`/`ffprobe` (basename-only, so no zip-slip), capped at 300 MiB per binary (decompression-bomb guard). Pinning per-asset SHA-256 is noted in the diff --git a/docs-developers/server/media.md b/docs-developers/server/media.md index c6761fc9..eac3db4a 100644 --- a/docs-developers/server/media.md +++ b/docs-developers/server/media.md @@ -313,7 +313,8 @@ first full pass the runner only reads books whose art is new. ## `DirectPlayable`: when a client should transcode -The scanner records each book's audio codec (ffprobe `codec_name`, verbatim) +The scanner records each book's audio codec (ffprobe `codec_name`, verbatim, +or the [in-process probe](scanner.md#the-in-process-probe)'s identical answer) in `books.codec`. `media.DirectPlayable(format, codec)` reports whether mainstream browsers can play the **stored** file natively. A **container** no browser plays (`containerFormats`: `mka`, by the file extension in `books.format`) never is, diff --git a/docs-developers/server/overview.md b/docs-developers/server/overview.md index 48e5a0a7..51a90ba3 100644 --- a/docs-developers/server/overview.md +++ b/docs-developers/server/overview.md @@ -162,7 +162,10 @@ once while the index is written in order). See [Scanner](scanner.md). ### `internal/metadata` Metadata extraction: embedded tags in-process via `dhowden/tag`, durations / -chapters / codec via ffprobe when available (`probe.go`), the release +chapters / codec when ffprobe is available, read in-process for MP3, MP4, FLAC, +Ogg and MKA by the configured ffprobe's release's rules (`fastprobe*.go`, the +[in-process probe](scanner.md#the-in-process-probe)) and by ffprobe itself for +any file the probe is unsure of (`probe.go`), the release date (`ReleaseDate`, a date tag as `YYYY[-MM[-DD]]`), and `DeriveFromPath` - the structural path heuristic (`Author/Series/01 - Title.m4b`; a lone folder above the book is its author, @@ -428,6 +431,9 @@ Every feature ships with a test (see `newTestCatalog`), `internal/web/web_test.go`, and so on. - A few scanner tests need `ffprobe` on the machine; without it they `t.Skip` (CI installs ffmpeg). +- The in-process probe is checked against ffprobe's answers over a fixture + corpus; see + [Gates and CI](../contributing/gates-and-ci.md#the-in-process-probes-fixtures). - **Security-critical code requires both an allowed and a denied regression test** - the enumerated list is in [Auth & security](auth-and-security.md#the-allowed--denied-test-rule). diff --git a/docs-developers/server/scanner.md b/docs-developers/server/scanner.md index 533a9de5..61fdd62c 100644 --- a/docs-developers/server/scanner.md +++ b/docs-developers/server/scanner.md @@ -306,8 +306,8 @@ flowchart TD F -- no --> G["detectMoves
fingerprint-match vanished → new paths,
MoveDurableState (logged)"] G --> H{"per book:
mtime+size unchanged
and probe data present?"} H -- "yes (skip; backfill has_cover /
suspect_parts if unset)" --> H - H -- no --> I["enrich: path heuristic + tags/ffprobe,
chapters, cover, fingerprint,
read problems, suspect parts"] - I --> J["catalog.UpsertBook (one tx):
scanned values + files + chapters,
then enrichment + overrides layered on, FTS"] + H -- no --> I["enrich: path heuristic + tags + probe
(in-process, ffprobe the fallback),
chapters, cover, fingerprint,
read problems, suspect parts"] + I --> J["catalog.UpsertBooks (up to 64 books a tx,
a savepoint each): scanned values + files + chapters,
then enrichment + overrides layered on, FTS"] J --> H H -- done --> HC["SetHasCover + SetSuspectParts
noted backfills, one tx each"] HC --> P{"discovery saw
the whole tree?"} @@ -386,7 +386,8 @@ several files at a time, and writes the index in one place (`library/scanread.go`, server#242): - **Slots.** `readSlots` bounds how many audio files are read at once, each - read (tags, then ffprobe: `metadata.ExtractTimeout`) holding a slot. There + read (tags and the [in-process probe](#the-in-process-probe), then ffprobe + for a file the probe leaves to it: `metadata.ExtractTimeout`) holding a slot. There are `scan.concurrency` of them, by default one per CPU up to `DefaultScanReaders` (8); see [Configuration](configuration.md#library-scans-scan). - **Books read ahead.** Each book to index is read (`enrich`) on its own @@ -394,14 +395,33 @@ several files at a time, and writes the index in one place (`readFiles`), up to a window of twice as many books as slots ahead of the next one written. - **One writer, in order.** The scan's own goroutine writes each book once its - read is done, in discovery order (`writeNext`), through - `catalog.UpsertBook`, on the one writer connection. The rows, their - ids, the file order, the chapter offsets and the scan's log come out as a - scan reading one file at a time writes them. -- **Stopping.** A Stop (or the scan's time limit) ends the ffprobe runs through - the scan's context. The reads under way are waited for and dropped (a read cut - short would record a false [read problem](#read-problems)), nothing more is - written, and the backfills read so far are kept. + read is done, in discovery order (`writeNext`), on the one writer + connection. The rows, their ids, the file order, the chapter offsets and the + scan's log come out as a scan reading one file at a time writes them. +- **Several books a transaction.** The books read wait in a batch and are + written together by `catalog.UpsertBooks`, since a transaction's begin and + commit cost about as much as a book's own writes (0.6 of 1.5 ms a book on a + synthetic library, 1.3 of 3.9 ms on a real one). + - *When a batch is written:* once it holds `writeBatch` (64) books, once the + first has waited `writeBatchAge` (250 ms), when the writer would otherwise + wait on a read, or when the scan ends or stops. Since the writer never + waits on a read while holding books, a scan whose reads are slower than its + writes (ffprobe on every file) still writes each book as its read is done. + - *How long it holds the writer:* the same 250 ms is `UpsertBooks`' budget. + A transaction takes another book only while it has run for less than that, + so a batch holds the writer (and every other write, a listener's progress + among them) no longer than that plus a book. + - *A failure:* each book is written under its own savepoint, so one that + fails is rolled back alone and logged while the rest of the batch is + written. A transaction that fails as a whole (its commit, say) has its + books written again one at a time (`UpsertBook`). +- **Stopping.** A Stop (or the scan's time limit) ends the in-process probes + and the ffprobe runs through the scan's context. The reads under way are + waited for and dropped (a read cut short would record a false + [read problem](#read-problems)), as is a finished read the writer had not yet + taken into the batch. The books already in the batch are written (their reads + finished before the Stop), nothing more is read, and the backfills read so far + are kept. - **On-demand indexing** ([`IndexPath`](#on-demand-indexing-indexpath)) reads with one slot, a file at a time on the request's own goroutine. - The skipped books' cheap backfills (above) still run one at a time on the @@ -440,13 +460,19 @@ doesn't need: when that is set lower (1 lists each folder as the walk reaches it). A higher `scan.concurrency` doesn't raise it: these calls wait on round trips, not CPUs. -- **`added_at` only for a new book.** Discovery reads no birth times. - `added_at` is set when a book is first indexed and never updated, so `enrich` - stamps it (`stampAddedAt`: the earliest `addedAt` of the book's audio files) - only for a book the index doesn't hold yet. A book that is re-indexed keeps - its row's value, so the scan sets a placeholder for it and reads none of its - files' birth times. On Linux a birth time is a `statx` per file on top of the - walk's `lstat`, so a no-change scan makes none. +- **`added_at` only for a new book, from discovery's `lstat`.** `added_at` is + set when a book is first indexed and never updated, so `enrich` stamps it + (`stampAddedAt`: the earliest `addedAt` of the book's audio files) only for a + book the index doesn't hold yet. A book that is re-indexed keeps its row's + value, so the scan sets a placeholder for it and reads none of its files' + birth times. The new book's files get no second `lstat`: `statAudio` keeps + what each file's `lstat` already told (`Book.FoundAdded`, from + `statEntry.found`). On Darwin that carries the birth time, so a file costs no + system call at all; on Linux, whose stat struct has no birth time, one + `statx` per file reads it, and the walk's mtime stands in where the + filesystem records none; on any other platform the walk's mtime is the + time, with no system call. A file `statx` finds gone is left to an `lstat` of + its own. A no-change scan makes no `statx`. - **A damaged file isn't probed again.** Without this, `problemCleared` would run ffprobe on a `probe_failed` book's file on every scan. When ffprobe rejects a file's data (`metadata.ErrProbeRejected`: its last stderr line ends in @@ -736,9 +762,11 @@ embedded data winning where it is trustworthy: scanned `genres`: split on `;` and NUL (never `,`, which a store's category name can hold), each trimmed and kept once in any case, with the placeholders `audiobook(s)`, `audio book(s)`, `speech`, `spoken word` and `other` dropped; ffprobe's `genre` replaces dhowden's when - it gives any), then ffprobe (when configured) for duration, chapters, the audio - `codec` (`codec_name` of the first audio stream - this is what feeds the - `direct_playable` API flag), and richer container tags. + it gives any), then the probe (when ffprobe is configured) for duration, + chapters, the audio `codec` (`codec_name` of the first audio stream - this + is what feeds the `direct_playable` API flag), and richer container tags: + the [in-process probe](#the-in-process-probe) for most files, ffprobe + itself for the rest, with the same answer either way. A **series tag** that carries its own position (the series atom, the MP4 `\xa9grp` grouping, `show` or `TXXX:SERIES`), as Audible-sourced tooling writes `Discworld, book #40`, is split into the series and its position @@ -791,7 +819,90 @@ path-derived metadata still works, durations fall back to chapter ends or remain 0, and `codec` stays empty - which the API treats as directly playable (`direct_playable: true`, so the web player streams the file as it is rather than through the transcoder). See -[Media & streaming](media.md). +[Media & streaming](media.md). With ffprobe disabled (`--ffprobe ""`) the +[in-process probe](#the-in-process-probe) is off too, so the server reads tags +and paths only (probing in-process with no ffprobe is a possible follow-up). + +### The in-process probe + +A first scan used to be mostly ffprobe: about 94% of a real library's read time, +almost all of it starting the process (some 31 ms on an Intel N305 before +ffprobe read a byte). So the scanner reads ffprobe's answer itself for the +common formats (`internal/metadata/fastprobe*.go`, server#261) and runs ffprobe +only for a file it can't be sure of. In the spike, its prototype took a scan of +a 1,165-book, 9,165-file slice of a real library from 197 s to 9.9 s, and 3 +of those files went to ffprobe. + +**One open, one cache.** `metadata.ExtractTimeout` opens a file once and reads +its tags (`dhowden/tag`), the iTunes narrator and the probe through one 64 KiB +block cache (at most 4 MiB kept), then closes it before any ffprobe run. + +**What it answers.** Exactly what the scan took from ffprobe: the format +duration as ffprobe's `"%f"` string, the first audio stream's `codec_name`, +the chapters, and the format tags `applyProbe` reads (`probeTagKeys`; a key +`applyProbe` starts reading joins that list). It reads MP3, M4B/M4A/MP4, FLAC, +Ogg Vorbis, Ogg Opus, Ogg FLAC and MKA, each reader a port of FFmpeg's demuxer +(an MP4's duration is the least of `mdhd`, `stts` and the edit list; a `chpl` +list or a chapter track, as FFmpeg picks; ID3 frames in FFmpeg's dictionary +order and overwrite rules; a Vorbis stream's start; Matroska's `TagDefault`). +Every other format goes to ffprobe. + +**Parity is the contract.** Every value equals what the configured ffprobe +gives, by the rules of its FFmpeg release (`ffRules`), so the probe changes no +stored duration, chapter or tag, and an upgraded server needs no re-index. +Which releases it runs for: + +- **7.1 and 8.1** (`knownRules`; they differ in MP3 gapless durations and in + empty UTF-16 ID3 strings). The official and LinuxServer.io images are both + Alpine 3.24, whose ffprobe is 8.1.3. +- **Any other release turns the probe off**, and ffprobe reads every file: + Ubuntu 24.04's 6.1, a master or nightly build (its version is no release, so + the BtbN builds the server + [downloads](configuration.md#ffmpegffprobe-resolution--auto-download) for + Linux and Windows get no speedup), or an ffprobe whose `-version` fails. + Tags are still read in-process either way. + +The release is read once per ffprobe path per process from `ffprobe -version` +(10 s at most), so an ffprobe upgraded in place is read again only after a +restart: its major.minor, so `7.1.1`, `7.1.1-3ubuntu1` and `n8.1.3` are +7.1, 7.1 and 8.1. + +**What goes to ffprobe, per file and silently.** A reader returns `unsure` for +any shape it does not model exactly, and that one file is read by ffprobe, +with nothing logged: + +- a VBR MP3 with no Xing or VBRI header (ffprobe estimates its duration from + the bitrate; keeping that estimate keeps the stored duration the same), and + a concatenated MP3; +- a FLAC with a CUESHEET block, or a FLAC or Ogg with an ID3 prefix; +- a fragmented or compressed MP4, or one with an unknown handler or sample + entry; +- a multi-stream Ogg; +- an MKA with no duration or a second SeekHead; +- another container named `.mp3` (WAV, FLAC, Ogg, MKV, MP4); +- a crafted table past a bound (a chapter track over 65,536 samples or with + counts past its tables, Matroska SimpleTags over 4,096 or 8 deep, a + compressed ID3 frame over 16 MiB); +- a read error, a reader's panic, no duration, or a probe limit (below). + +**Limits.** A file's probe stops after 5 s or 64 MiB read, and that file goes +to ffprobe. The deadline is the caller's context and +`min(timeout, maxProbeTime)`, under the time `ExtractTimeout` gives ffprobe +(30 s, see [Read problems](#read-problems)); `maxProbeBytes` is far past the +real library's p95 of about 0.9 MiB. Both are checked on every block read, before an ID3 frame is inflated or a repeated +Vorbis key grows (`blockFile.check`), and every 256 ticks in the walks that +loop without reading (`blockFile.tick`, a `probeLimit` panic `fastProbe` +recovers). + +**A file neither reads** keeps ffprobe's error as its `ProbeErr`, so Health +shows the same `probe_failed` text as before (`ErrProbeRejected` included). + +**Keeping parity.** A new FFmpeg release that changes a rule would change a +value, so the probe is checked against ffprobe mechanically: a fixture corpus +with each known release's answers, a live comparison and a CI job (see +[Gates and CI](../contributing/gates-and-ci.md#the-in-process-probes-fixtures), +which also says how to add a release). `cmd/probediff ` compares the probe with the installed +ffprobe over any folders and times both: the tool to point at a real library. ## Chapter normalization @@ -976,7 +1087,7 @@ reading a book's files, in three columns: |---|---| | `unreadable` | a file couldn't be opened (`metadata.Metadata.OpenErr`) | | `empty_file` | a file is 0 bytes | -| `probe_failed` | ffprobe is configured and couldn't read a file (`ProbeErr`) | +| `probe_failed` | ffprobe is configured and couldn't read a file (`ProbeErr`): the in-process probe left it to ffprobe, which failed | `scan_error_file` is the library-relative file and `scan_error_detail` the OS's or ffprobe's own message, without the absolute path (the console shows the file @@ -998,7 +1109,9 @@ server restarts (see [A rescan that finds nothing changed](#a-rescan-that-finds-nothing-changed)). **A probe that ran out of time.** Each ffprobe run gets 30 seconds -(`metadata.ProbeTimeout`); one that runs out is killed, its `ProbeErr` wraps +(`metadata.ProbeTimeout`; the in-process probe's own bound, 5 s, only sends a +file to ffprobe, see [The in-process probe](#the-in-process-probe)); one that +runs out is killed, its `ProbeErr` wraps `metadata.ErrProbeTimeout`, and the problem is `probe_failed` with the detail `ffprobe timed out after 30s`. Files sharing a slow disk or a busy share can each take longer than that while [others are read beside diff --git a/docs-users/admin/libraries.md b/docs-users/admin/libraries.md index 353bc1e3..9a4283cd 100644 --- a/docs-users/admin/libraries.md +++ b/docs-users/admin/libraries.md @@ -175,6 +175,14 @@ variable) and restart. It isn't in the console; see the cancel a waiting one, and keeps a history of every scan with its log (see [Jobs](health.md#jobs)). +**With FFmpeg 7.1 or 8.1, a first scan is many times faster** (from 2.5.1). The +server then reads the lengths and chapters of most MP3, M4B, M4A, MP4, FLAC, +Ogg, Opus and MKA files itself instead of starting ffprobe for each one, and +asks ffprobe only about a file it can't be sure of (an unusual MP3 or MP4, for +example). The official and LinuxServer.io Docker images have 8.1. With any +other version, ffprobe reads every file. See [ffmpeg and ffprobe](../getting-started/install-binary.md#ffmpeg-and-ffprobe) +for how to check yours. + Scans also run automatically: - for every library **when the server starts**, diff --git a/docs-users/admin/server.md b/docs-users/admin/server.md index 4c8d0917..d1ccb13d 100644 --- a/docs-users/admin/server.md +++ b/docs-users/admin/server.md @@ -320,6 +320,10 @@ How it behaves: **Not available** with what that means. ffmpeg converts formats browsers can't play to MP3 while someone listens, and copies the MP3 out of `.mka` (Matroska) files so every player can play them; ffprobe reads lengths, chapters and codecs. +When ffprobe is version 7.1 or 8.1, the server reads most files' lengths, +chapters and codecs itself during a scan, much faster, and uses ffprobe for the +rest; with another version, ffprobe reads every file (see +[ffmpeg and ffprobe](../getting-started/install-binary.md#ffmpeg-and-ffprobe)). The MP3 copies are kept in the data folder, under `cache/remux`, so each book is copied once rather than on every play. The folder holds at most 8 GiB: past diff --git a/docs-users/faq.md b/docs-users/faq.md index b25e8d2f..16f9a415 100644 --- a/docs-users/faq.md +++ b/docs-users/faq.md @@ -36,8 +36,12 @@ see [Organizing your library](./getting-started/organizing-your-library.md). No - but you'll want it. The server uses two companion tools: - **ffprobe** reads durations, embedded chapters, and each file's audio codec. - Without it, books still index and play (titles and authors come from tags - and folder names), but durations and chapter lists are unavailable. + With FFmpeg 7.1 or 8.1, the server reads most files itself during a scan, + which is much faster (see + [ffmpeg and ffprobe](./getting-started/install-binary.md#ffmpeg-and-ffprobe)). + Without ffprobe, books + still index and play (titles and authors come from tags and folder names), + but durations and chapter lists are unavailable. - **ffmpeg** enables on-the-fly conversion of browser-unfriendly codecs to MP3, which the web player uses automatically for such books. Without it, that feature is simply off, and those books play only in the mobile apps diff --git a/docs-users/getting-started/install-binary.md b/docs-users/getting-started/install-binary.md index 06e1358d..66844657 100644 --- a/docs-users/getting-started/install-binary.md +++ b/docs-users/getting-started/install-binary.md @@ -29,6 +29,8 @@ AudioSilo uses ffmpeg/ffprobe for durations, chapter extraction and on-the-fly t In short: on the common path there is nothing to install, and if the download can't happen, nothing breaks. +**For faster first scans, use FFmpeg 7.1 or 8.1** (from 2.5.1). With one of those versions the server reads most files' lengths and chapters itself, without starting ffprobe for each one, and asks ffprobe only about the files it can't be sure of. With any other version (for example Ubuntu 24.04's 6.1), every file is read by ffprobe, which works but makes the first scan of a big library much slower. The copy the server downloads for Linux and Windows is a development build, not a release, so it gets no speedup either: install a 7.1 or 8.1 package and restart the server, which then finds it first. Run `ffprobe -version` to see which you have. ffprobe is still needed either way. + ### Run it from a terminal ```bash @@ -52,7 +54,7 @@ Useful flags: | `--data ` | Where config, database and certificates live (default `./data`). | | `--setup` | First run prints a one-time link to choose the admin password in the browser, instead of printing generated credentials. | | `--ffmpeg ` | Explicit path to ffmpeg; `""` disables transcoding and the MP3 copies of `.mka` books. | -| `--ffprobe ` | Explicit path to ffprobe; `""` disables duration/chapter extraction. | +| `--ffprobe ` | Explicit path to ffprobe; `""` disables duration/chapter extraction (the server then reads none itself either). | ### Linux packages and systemd diff --git a/docs-users/getting-started/organizing-your-library.md b/docs-users/getting-started/organizing-your-library.md index 814f468b..51bc8114 100644 --- a/docs-users/getting-started/organizing-your-library.md +++ b/docs-users/getting-started/organizing-your-library.md @@ -88,7 +88,7 @@ For each book, AudioSilo starts with the names on disk, then lets the audio file - **From your folders**: the book's folder (or file) name becomes the title; a leading number like `03 - The Title` (but not one that runs into the name, like `3rd Rock` or `20,000 Leagues Under the Sea`) is read as its position in a series, decimals included, so a novella between books 2 and 3 named `02.5 - The Title` is number 2.5 (from 2.5.0; an older server reads it as 2); the parent folder is read as the series and the folder above that as the author. A book with only one folder above it (`Author/Title/`) takes that folder as its author, with no series. So an `Author/Series/03 - Title/` or an `Author/Title/` layout works with no tags at all. - **From embedded tags**: a title (audiobooks usually carry it in the *album* tag; when the album holds the series name and the *title* tag carries the full "Series: Book" title, the more specific title tag wins), the author (album artist or artist tag), the narrator (the narrator tag, which Audible and most audiobook tools write into an M4B, or the composer tag when there is none), the series where a tag exists (a series tag that ends in a marked number, like the grouping tag "Discworld, book #40", gives the series Discworld and its position, 40; a name that only ends in a number, like "Metro 2033", stays whole), and the genres in the *genre* tag (several in one tag are split by `;`, never by a comma, which a store's category such as "Mystery, Thriller & Suspense" holds; a genre that only says the file is an audiobook, such as "Audiobook", "Speech", "Spoken Word" or "Other", is left out). Junk tag titles like "Track 01" are ignored in favour of the folder name. - **From Audiobookshelf's `metadata.json`**, when the book's folder has one: its title, subtitle, authors, narrators, series, published year, description, publisher, language, abridged, genres, tags, ASIN and ISBN go over the tags. See [Coming from Audiobookshelf](#coming-from-audiobookshelf). -- **From ffmpeg's ffprobe**, when available: durations and any chapter list embedded in the file, so a chaptered M4B shows its real chapters in the player. +- **From ffmpeg's ffprobe**, when available: durations and any chapter list embedded in the file, so a chaptered M4B shows its real chapters in the player. With ffprobe 7.1 or 8.1 the server reads most files itself, with the same results, much faster (see [ffmpeg and ffprobe](./install-binary.md#ffmpeg-and-ffprobe)). - **From the community database**, for chapters only: a book whose files have no chapters of their own (one long file, or a folder with one chapter per file) gets the community's chapter list once it is [matched](../admin/books.md#matching-with-community-metadata) and the list fits your copy. A book with only broad chapters can be switched to the community's more detailed ones. See [Community chapters](../admin/books.md#community-chapters). :::note[Narrators of books already on your server]