diff --git a/CLAUDE.md b/CLAUDE.md index 2dc9df90..f265f167 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -270,1754 +270,51 @@ future metadata site can attach enrichment without reshaping the schema. The `books` metadata columns are the effective values (scan, then an Audiobookshelf `metadata.json`, then enrichment, then admin overrides; see Metadata overrides below). -## Conventions - -- **Every feature ships with a test.** Handler/integration tests use the - `newTestEnv` harness in `internal/api/api_test.go` (a temp-file SQLite, so reads - go through the read-only reader pool as in production and a write sent through a - read method fails the test, + `testdata/library` fixtures; `catalog`'s - `newTestCatalog` is file-backed for the same reason). Never migrate per test: open a test database with - `storetest.Open(t)` (`internal/store/storetest`; `storetest.Path(t)` for a file a test opens itself), not - `store.Open` on a new file or `:memory:`. Under `-race` running every migration costs about a second, so it - migrates once per test binary and copies the file per test (file-backed, so reads use the reader pool). - A package creating many password users calls `auth.UseCheapHashingForTests()` from `TestMain` - (`internal/api`, `internal/auth` and `internal/matchrun`'s `main_test.go`; a real-cost argon2id hash is ~0.25 s under `-race`; it panics outside a test - binary). A new top-level test starts with `t.Parallel()` (each has its own database and - temp dirs; the only thing tests share is the read-only `testdata/` fixtures, so copy one into - `t.TempDir()` before changing it: `copyFixtures` in `internal/api`) unless it calls `t.Setenv` or `t.Chdir` - (which panic in a parallel test) or writes a package-level var (pass the value in instead, as - `hashWithCost` does for `TestHashCost`). Pure-logic tests sit next to the code (see - `internal/api/middleware_test.go`, `internal/catalog/shares_test.go`, - `internal/web/web_test.go`). **Security-critical code requires both an allowed - and a denied regression test** - anything touching `library.SafeJoin`, - `Scope.Allows`/`VisibleInBrowse`/`pathFilterSQL` (its single-parameter form too), the household member rule - (`catalog/members.go`) and the member routes, the rate limiters, - `auth.ResolveRequest`/`lookupToken`, the cast token checks (`relpath.Within`/`Clean`, `auth.castGrant.covers`, - `ResolveCastRequest`, `api.authenticate`/`castOwnBook`), or `web.htmlCSP`. Keep business logic in the non-`api` - packages so it stays unit-testable (`api` is transport-only). -- **Migrations are append-only**: add `internal/store/migrations/000N_*.sql`; - never edit an applied migration. Applied names are tracked in `schema_migrations`. -- **Secrets** (tokens, auth codes) are stored only as SHA-256 hashes; passwords - use argon2id (`auth/hash.go`). Never log or persist plaintext secrets; the - first-run banner prints them once and is the only place they appear. -- **Connect / invite / pairing flow** (`internal/web` connect page + `api/qr.go`): - the admin's **Copy invite** button mints an auth code and shares - `/connect#code=...` - the code rides in the URL **fragment** so it never - reaches the server or its logs. The connect page auto-redeems a fragment code, - showing a QR plus **Open in app** / **Open web player** buttons. `buildPairing` - emits two carriers for the pairing token: `web_url` - (`/web/connect?token=` - encoded in the QR; opens the app via a Universal/ - App Link when the domain is claimed, else the embedded web player) and `uri` - (`audiosilo://connect?...` - custom scheme, launches an installed app on any - domain). Both end with `sid=` (after `home=`/`away=`; also the payload's - `server_id`), so an app falling back to the link's http home checks `GET /server`'s - `server_id` there before sending the token in cleartext. **The QR is as redeemable as the invite**: a pairing token minted by - redeeming a code is linked to it (`tokens.auth_code_id`) and inherits the - invite's uses/expiry, so each device being set up can scan the same QR (the - redeem response's `uses_remaining`/`code_expires_at` advertise the budget); - recovery-derived tokens instead last `auth.recoveryPairingTTL` (multi-scan - within it), and - `/auth/pair` + demo tokens stay unlinked/single-use (the console's book page - "Listen in player" mints one with `/auth/pair` and opens the same-origin - `/web/connect?token=&sid=&uid=&book=&path=` in a new tab, never the payload's - `web_url`, whose base may be another origin). Invite codes minted via - the admin API default to 5 uses / 1-day expiry (`defaultAuthCode*` in - `handlers_admin.go`); explicit values override. -- **Home and away addresses (`addresses` capability)**: a server - has an **away** address (`public_url`, works from anywhere) and may have a **home** one - (`lan_url` / `AUDIOSILO_LAN_URL` / console `general.lan_url`, live; else derived from the - request's own `Host` when that is a home-network host). The pure core is - `config.Addresses(scheme, host)` + `config.isHomeNetworkHost` (private RFC 1918 / ULA - `fc00::/7` / link-local IPs, `.local` / `.lan` / `.home.arpa` / `.internal` names, single-label names; - NOT loopback, `localhost` or CGNAT `100.64.0.0/10`); a home equal to the away is dropped; - `X-Forwarded-*` is not trusted (like `baseURL`), and a proxied request (any forwarding - header) derives no home (its Host is the proxy's upstream). `api.addresses(r)` wraps it. It rides on - every pairing payload (`addresses`, nil when both empty) and as `home=`/`away=` params - APPENDED after the existing ones on `uri` and `web_url` (only when non-empty), on the exchange / login / demo session - envelopes (`sessionEnvelope`), and at `GET /addresses` (any signed-in user, `{}` when - neither) so an already-paired device learns a newly set address. Native apps probe the - home address and check its `server_id` before sending a token there. -- **Home HTTP (`tls.home_http`, security-relevant)**: the apps can't trust a self-signed - certificate, so on `tls.mode: selfsigned` with `home_http` on (the default; - `AUDIOSILO_TLS_HOME_HTTP`, console `network.tls_home_http`, restart; `config.ServesHomeHTTP`) - the bind port answers both protocols. `server.splitListener` peeks each connection's first - byte in its own goroutine (bounded by the 10 s header timeout; Close drops a peeking - conn): `0x16` becomes a `*tls.Conn`, anything else stays plain, and ONE `http.Server` - serves both (ALPN `h2` set by hand, as `ServeTLS` would). `server.homeHTTPGate` serves a - plain request only when the connection's own `RemoteAddr` (never `X-Forwarded-For`) is - loopback, or `config.HomeNetworkIP` (private/ULA/link-local; not CGNAT) AND the Host is - `config.HomeNetworkHost` (`homeRequest`: a NAT/proxy gateway's private source, Docker's - IPv6 proxy say, still carries the internet client's public Host); anyone else gets a 308 - to the same URL on https (`:80` for a port-less Host, as `baseURL`; 400 for an unusable - Host). Off whenever `tls.cert_file` is set (`ServesHomeHTTP`: a supplied cert may be trusted). - `config.Addresses` then derives the home address as `http://` even for an https request, - keeping the request's port (`:443` added when the Host has none, the port that answers - http); a configured `lan_url` is used as is, and `base_url`/`web_url`/`uri server=` are - always https (`api.baseURL`: a plain-HTTP request gets the https base on its port, 80 for a - Host without one, `config.HostWithPort`; a proxied one keeps its port-less Host, https's 443), so a link copied from a console opened over http stays https. No HSTS on self-signed and no `upgrade-insecure-requests` in any - CSP (`TestCSPAllowsPlainHTTP`), so the console and player work over http on the LAN. -- **Invite vs recovery (`auth_codes.kind`)**: an auth code is either an admin-minted - `invite` (bounded) or a user-owned `recovery` code (durable: unlimited uses, never - expires). Both pair through the same `ResolveAuthCode` → `IssuePairingToken` → - `ConsumePairing` path. **Redeem validates without consuming** (opening an - invite link costs nothing); **exchange claims the use** - `ConsumePairing` - folds the cap check, the code-expiry check and the first-claim `redeemed_at` - stamp into one atomic UPDATE, and rejects a disabled/deleted user first, so a - rejected attempt never burns a use or marks an invite accepted (`uses` counts - devices that actually paired). Linked pairing tokens die with their code: delete/ - supersede/recovery-regeneration cascade (`ON DELETE CASCADE`), rotate revokes - them explicitly. Recovery decouples - re-auth from invitation: a signed-out/password-less user mints a recovery code from - the player's Settings (`POST /auth/recovery`) and re-pairs without an admin. Recovery - mint/redeem and `POST /auth/password` are gated by `accountLimiter` and **refused for - demo accounts** (`User.IsDemo`) so a throwaway session can't forge a durable login. - `ListAuthCodes` returns only invites; recovery presence surfaces as - `User.HasRecovery`, and the admin can revoke a leaked one via `DELETE - /admin/users/{id}/recovery` (`ClearRecoveryCode`) - the only lever, since recovery - codes aren't listable. **Invite hygiene**: `CreateInvite` mints and, in one - transaction, supersedes the user's other *still-redeemable* invites - (`supersedeActiveInvites` - not expired, not used-up) so there's exactly one active - invite each; spent/expired ones stay as history. `POST /admin/authcodes/{id}/rotate` - (`RotateAuthCode`) regenerates an invite's secret in place (the console's "Rotate"), - **preserving** its `max_uses` and renewing its expiry for the original window (never - silently downgrading to defaults) and **revoking the invite's outstanding pairing - tokens** (a QR already on screen dies with the old secret) and, like a mint, retiring - the user's other still-redeemable invites; `redeemed_at` records - acceptance (first successful exchange) but the console buckets invites by whether - they are still redeemable, not by `redeemed_at`. **Self- - service password**: `POST /auth/password` reuses `SetPassword`; setting a first - password needs no challenge, but changing an existing one requires `current_password` - (`CheckPassword`), an empty password is rejected (clearing is admin-only), and the - admin-must-keep-a-password guard still holds. -- **Personal API keys (`tokens.kind='api'`)**: user-minted, **non-expiring** bearer - tokens for headless integrations, owner-scoped mint/list/revoke via - `POST`/`GET`/`DELETE /auth/tokens` (`IssueAPIToken`/`ListAPITokens`/ - `RevokeTokenByID`; label carried in `device_name`, ≤100 chars). `requireAuth` now - accepts **session OR api** (`ResolveRequest(..., KindSession, KindAPI)`), so a key - authenticates like a session acting as its owner (an admin's key passes - `requireAdmin`; media `?token=` accepts it too) but is never valid for pairing - `/auth/exchange`; a pairing token is never accepted as a bearer credential. - Secrets are stored SHA-256-only and shown once; create/list/revoke go through - `gateSelfService` (shared `accountLimiter` + demo refusal). **Containment**: a key - authenticates as its owner but cannot mint a *fresh durable credential* - - `ResolveRequest` returns the matched kind (`Credential.Kind`) and `denyAPIKey` refuses an api-key - caller (403) on `POST /auth/{tokens,recovery,pair,password}`, so revoking a leaked - key cuts off everything it could reach (it cannot spawn another key/recovery - code/pairing token/password that outlives its own revocation - GitHub's "a token - cannot create tokens"). It may still list/revoke keys and clear a recovery code - (those only reduce access). Surfaced by the `api_keys` capability. -- **Web player at `/web`** (`web.go`, served from `cfg.WebDir`): a separate Expo - Router project (`~/dev/audiosilo/audiosilo-frontend`) exported as a static site. It is - **not vendored** in this repo or the binary - the server serves it at runtime - from `web_dir` (env `AUDIOSILO_WEB_DIR`), which the Docker image bakes in at - `/app/web` from a pinned prebuilt frontend image (see `Dockerfile`). Empty - `web_dir` → `/web` is unmounted and the `web_player` capability is false. The - export must be built with `baseUrl=/web` (frontend `app.json experiments.baseUrl`) - so asset URLs resolve under the subpath. The handler resolves per-route HTML, - falls back to `index.html` for client-routed deep links, 404s missing assets, - and sets a **scoped CSP** per HTML response (strict `script-src`: `'self'`, the Cast Web - Sender's three scripts on `https://www.gstatic.com` by PATH (`web.castSenderScripts`: - `/cv/js/sender/v1/cast_sender.js`, `/cast/sdk/libs/sender/1.0/cast_framework.js`, - `/eureka/clank/`; capability `cast`; the ONLY remote sources in the policy, never the whole - origin, which also serves AngularJS, a CSP-bypass gadget; `TestHTMLCSPCastSender` pins the - whole string) and a sha256 - hash of that doc's inline scripts; `style-src` allows `'unsafe-inline'` for - react-native-web's runtime styles). Admin/connect pages keep the stricter - site-wide CSP. Compatibility is by construction (the image pins a matching web - build); native apps negotiate via `GET /server` capability flags. -- **Admin console (`admin-ui/` + `internal/web/adminui`)**: a React SPA in the Shelf design - (workspace `ADMIN-CONSOLE-PLAN.md` holds the phases still to come; `admin-ui/STYLEGUIDE.md` the - design). It replaced the classic vanilla-JS console at the Phase 1b cutover (no switch, no - `/admin/classic`). Vite builds into `internal/web/adminui/dist`, embedded with - `//go:embed all:dist` (only `dist/.gitkeep` is committed; a build without it serves a 503 - "console not built" page). Serving goes through **`internal/web/spa`**, the one SPA handler - the console and the web player share: files under the asset dirs are immutable and 404 when - missing, other files and HTML revalidate, a missing top-level file with an extension 404s, - anything else boots `index.html` for client routing; MIME types are pinned process-wide. - Every static file (console, player, connect/setup assets) carries a strong content-hash ETag - (a match is a 304, documents included) and text types go out gzipped to clients that accept it - (`spa.Files`: worked out once per file and kept, redone when a web_dir file's size or mtime - changes; never for a Range request; `Vary: Accept-Encoding`). - **The CSP does not change for it**: no inline script/style anywhere (`theme-init.js` is - external, Base UI runs under `CSPProvider disableStyleElements`, banned libraries are - ESLint-enforced, `admin-ui/scripts/check-csp.mjs` fails the build and `TestEmbeddedBuild` - checks the embedded `index.html`). It keeps the classic console's token key - (`localStorage["audiosilo_token"]`), so an admin stays signed in across the upgrade. A 403 from - an `/admin` endpoint makes the console re-check `/me` and sign a demoted admin out. - Admin-only endpoints that exist for it: `GET /admin/libraries` adds `book_count` and - `available` (root reachable: `Scanner.RootAvailable`, a 2 s-bounded, 15 s-cached, one-at-a-time - probe per root so a hung NFS mount can't stall the page, plus "empty while books are indexed" - and "last scan stopped at the guard"); scan status adds `unavailable` and `queued`, and - `API.startScan` queues the scan in the job queue (see "Job queue" below), which marks the - library queued before the request returns so the first poll sees it; `GET /admin/invites` lists every account's invites (`auth.ListInvites`, never a - code); `GET /admin/fs/dirs?path=` is the add-library folder picker (`library.ListDirs`: - absolute paths, folder names only, no dot-folders, 1,000-entry cap); `GET /admin/shares` adds - `member_ids` (`catalog.ShareMembers`). -- **Community metadata lookup (Phase 1.5, `internal/meta`)**: `GET - /api/v1/libraries/{id}/meta?path=` (authed, scope-checked via `authorizedPath` - + `bookForPath`, exactly like `item`) resolves the book's `asin`/`isbn` - (backfilled via `book_enrichment`) against the community metadata API - (`metaserve`, meta.audiosilo.app) and returns a composed enrichment envelope - (work + matched recording + series rails, each carrying its own `web_url`). - The `work` also carries the community **characters** and **recaps** (the CC - BY-SA expressive layer: spoiler-tagged, position-keyed - `reveal`/`through` - are logical work-chapter positions) when metaserve has them, plus a - whole-book `recap_summary` (`{in_short, ending}` - both summarize the whole - book, and `ending` is a full spoiler by construction); all three are additive/`omitempty`, mirrored on - `upstreamWorkDetail` and `MetaWork` and passed through by - `toCharacters`/`toRecaps`/`toRecapSummary` (which drops an all-blank summary). - **Work-id lookup**: `GET /api/v1/meta/work?id=` (authed, *not* - library-scoped - global read-only community data with no path to authorize; - the id is a query param because work-id slugs are not path-segment safe, and - `internal/meta` URL-escapes it upstream) returns `{"work": {...}}` with the - same `MetaWork` shape. It is the "catch me up on the previous book" path: a - series rail carries sibling work ids but no characters/recaps, so a client - resolves one of those ids here. `Service.Work` wraps the single - `works/{id}` GET and caches under a `"w:"` key space in the SAME bounded - cache, with Enrich's TTL policy (24h / 1h not-found / 2min error) and the - same "never cache a caller-cancelled failure" rule. Because the work id is the - one **caller-chosen** value that becomes a cache key, an outbound GET and a log - field, it is bounded on three axes: the handler rejects an id over - `maxWorkIDLen` (200 bytes) or carrying control characters (400 `invalid id`, - `validWorkID` - transport-level hygiene, not a slug grammar); the `"w:"` key - space has its own cache quota (`maxWorkEntries`) so an id flood can only evict - other work entries, never the enrichment cache; and uncached upstream fetches - go through a small semaphore (`maxConcurrentWorkFetches`, the transcodeSem - pattern) as an amplification bound on the shared community service. A - wrong-shaped upstream 200 (an id colliding with a literal metaserve route - decodes leniently to an empty work) is treated as `ErrNotFound`, never cached - or served as a positive blank work. Degradation: metadata off - -> 404; missing/blank `id` -> 400; malformed `id` -> 400; unknown work id -> - 404; upstream error -> 502. - Config is `metadata.{enabled,base_url,region,mode}` (env `AUDIOSILO_METADATA_ENABLED` / - `AUDIOSILO_METADATA_BASE_URL` / `AUDIOSILO_METADATA_REGION` / `AUDIOSILO_METADATA_MODE`; `base_url` must be an - absolute http(s) URL when enabled; `region`, the preferred Audible marketplace, one - of `config.Regions` or "", applies live; `mode` is `remote` or `mirror`, read at start, see Mirror - mode below) - one key disables ALL outbound calls (lookups and the mirror's downloads). **Runtime toggle**: `meta.Service` - is constructed in `api.New` whenever `base_url` is valid (`MetadataConfig.ValidBaseURL`), - regardless of `enabled`, and the live config's `metadata.enabled` gates it - so an admin - can flip it on/off without a restart (Server settings, below). The handler and the - `metadata` capability both gate on `metadataOn()` (`a.meta != nil && enabled`); - `a.meta == nil` (empty/invalid `base_url` at start) can't be enabled (400 - `metadata_unavailable`); a new `base_url` waits for a restart. `meta.Service` owns the compose logic - (lookup -> works/{id} -> pick the recording by `recording_id`, first as - fallback -> up to 3 series rails, one per ordering family) behind a bounded in-memory TTL cache (24h - positive / 1h not-found / 2min transport-error, ~2048-entry cap) so a hot path - or a down upstream isn't hammered; the api handler (`handlers_meta.go`) is - transport-only. Degradation: disabled -> 404 (and the `metadata` capability is - false, so clients hide the UI); no asin/isbn or no upstream match -> `200 - {"matched": false}` (with a `work` of the book's own when the Bridge generated - characters, recaps or a description for it: `meta.Answer`); upstream unreachable -> 502, unless - the book has generated content (this server's own), answered as unmatched with - it. **Generated description** (Phase 1c): a book's `description.json` rides in the additive - `work.generated_description: {text}` (`MetaGeneratedDescription`), NEVER in `community_description` - (a shipped 2.0 app shows that first and credits the community; it ignores the new field), and - `work.generated.description` (bool, always sent beside `characters`/`recaps`) marks it. It fills - only a work with no `community_description` (the core `description`, the publisher's kind, doesn't - block it: the player orders those). `fillable` (generated.go) is the one statement of which parts a - work leaves to fill (`metadata.AttachmentParts`): `Fillable` loads by it and `parts` lays in by it. - `generatedFor` reads only the parts it names that the book's signature names too - (`Book.GeneratedParts`), plus the provenance (`catalog.BookAttachments` with a parts mask: a work with - community characters and recaps reads just the description and its provenance, and none for a book - without one); `?spoilers=hide` keeps - it (spoiler-free by construction: the Bridge writes it from the opening chapters' notes only), and - attribution never counts it. `generated.generated_at` is bridge.json's `generated_at` (the - characters' and recaps' run), or its `description.generated_at` (`Generated.DescriptionAt`) when the - description is all the mark names: a description-only job keeps the run's time. An unmatched book - with only a description answers `{matched: false, work: {title, generated_description, generated: - {description: true, ...}, ...}}`. Out of scope for now: no cover remote-fallback, no tag-based ASIN extraction. - **Reading-order families** (metaserve schema_version 7): `seriesRails` collapses - each family (key `ordering_of || id`) into ONE rail whose top-level view is the - MAIN view - the ref with no `ordering_of` (the primary), else the first ref - so - a shipped player that ignores the new fields never sees a chronological order's - earlier books as "previous". The other orders ride along as additive - `orderings` (at most `maxOrderingAlternates` = 2 per family, failures make the - envelope partial), and `maxSeriesRails` counts FAMILIES. Every main view is - fetched before any alternate, so under `composeTimeout` a slow upstream costs - alternates, never rails (`TestEnrichMainsBeforeAlternates`). A pre-v7 upstream - yields byte-identical rails (`TestEnrichPreV7RailsUnchanged`), except that a - work listed at two positions of one series is now one rail rather than two - (`TestEnrichRepeatedMembershipIsOneRail`). Server-side because shipped players - lag. - **Series rules** (`meta/seriesrules.go`): which of a work's series a book of it - is in. Never a variant reading order (`ordering_of`), a translation of another of - the work's series (`translation_of`; unless that one's derived `language` differs - from the work's) or a series whose derived `language` differs from the work's - (primary subtags; unknown agrees); an EDITION - a series named - exactly like another of the work's plus a bracketed qualifier or a "narrated by" - tail ("Discworld (abridged)", "Harry Potter (Full-Cast Editions)") - only for the - recordings its qualifier names (`editionHolds`: abridged, full cast = >= 4 - narrators or one named "Full Cast", the named narrator; "(Unabridged)", a - language edition or an ordering note hold none; "narrated by" after a spaced dash - or a colon). Editions are read from the name - because the community API links no series to a recording. A name that only - extends another ("Discworld: Tiffany Aching") is a series of its own. A match - candidate's `series` is the work's series every recording is in, and each - recording's additive `series` the book's list as that recording, its own edition - first (`workSeries.of`), which `matchrun.Propose` and the match dialog's - `communityValues` read (so the console has no copy of the rule); series carry an - additive `id`. The facts come from `series/{id}`, each series once per request, - concurrently through `workSem`, at most `maxSeriesFacts` per match within - `seriesFactsTimeout` (none for a work they can't rule on: no language, one - series), cached in the `s:` key space (quota `maxSeriesEntries`); an unreadable - record drops nothing, and a repick (`MatchQuery.NoSeries`) reads none. Rails take - the rules too (`railMains`, before the `maxSeriesRails` cap): an edition named for - a kind of recording only for the recording the lookup named (or the work's only - one; never `pickRecording`'s first-recording fallback), then first (the book's main - series, as a match makes it), so the cap goes to the book's series (a Discworld novel in six series kept its Tiffany - Aching rail); other editions keep their place (pre-v7 rails unchanged); a series - the cache already knows is foreign is left out before the cap, and a main view - whose fetched record is foreign once read, handing its slot to the next family (at - most `maxForeignRails` times; a translation judged with every record read). - **Persistent meta cache** (`meta_cache`, migration 0024): `meta.Store` is a - SQLite second level behind the memory cache (`catalog/metacache.go`, adapted to - `meta.Store` by `api.metaStore`, which logs failures; best effort, a failure - never fails a lookup). Same keys and TTLs; persisted: enrichments (positive, - not-found, incomplete) and positive works only - never transport errors, never - work-id misses. A fresh row warms memory for its remaining TTL; a positive row - however stale is served when the upstream fails (not on caller cancellation), - held in memory for errorTTL. Rows carry `storeVersion` and the metaserve - `source`; others are ignored. Derived, rebuildable, not user state; the - launcher's retention keeps the newest `catalog.MetaCacheRows` (20 000), works - (`w:`, caller-chosen ids) within a share of their own (`catalog.MetaCacheWorkRows`, - 2 000) so they never push the books' enrichments out. A work's later 404 (remote mode; never in - mirror mode, see Mirror mode) - replaces its stored row (no new row for an unknown id). Writes are bounded by - `storeWriteTimeout` (250 ms): a busy writer costs the row, not the response. No config - key: only the service reads or writes it, so `metadata.enabled` off touches nothing. - **Mirror mode** (`metadata.mode: mirror`, env `AUDIOSILO_METADATA_MODE`, `remote` the default; read - at start, a `restart` setting): the server keeps a local copy of metaserve's own database and - answers the SAME questions in-process with metaserve's own code, so no book is looked up over the - internet. The copy is audiosilo-meta's data release artifact (`meta.sqlite.gz`, ~450 MB, ~1.8 GB - decompressed; the CC0 core and the CC BY-SA community layer in one file), and the module - `github.com/kodestar/audiosilo-meta` is a dependency: `pkg/release` (the ONE implementation of - the release asset contract: newest non-draft release carrying the asset, allowlisted GitHub hosts, - stall watchdog, gz digest verified while streaming, temp file + rename) and `pkg/query` (`Open`, - `MaxSchemaVersion`, and `NewHandler`: metaserve's JSON API handler, byte for byte). **Parity by - construction**: `meta.Service.SetMirror` swaps the client's transport for `fallbackTransport` - (`internal/meta/mirror.go`), which serves each request the client already builds through the local - handler (`meta.Mirror.Handler()`; an in-memory ResponseWriter, `localTimeout` 15 s, - `maxLocalBody`) while the copy is `Ready`; no copy, or a failed answer (a 5xx, e.g. a query - this code can't run on an artifact schema newer than `query.MaxSchemaVersion`; an answer too - large; a timeout; a panic), sends the request to `base_url` unchanged through remote mode's own - `*http.Client` (its `clientTimeout` covers the body); a failed answer is logged there, at most - every 10 min (`SetMirror(m, log)`). A 404 is the copy's answer; a retired slug's 301 is followed - on the same host, so it resolves locally. A `base_url` with a path is stripped before the copy's - handler and put back on a root-relative `Location`; a caller gone mid-answer is not a fallback. Everything above the client (compose, rails, match, the - caches) is unchanged and `meta_cache` rows keep `Source = base_url` (switching modes keeps the - cache warm). Two rules differ in mirror mode: the COPY's "no match" never replaces a stored POSITIVE - row in `Enrich` or `Work` (`keepStored`: the stored answer is served stale, held in memory for - errorTTL, the row untouched; a lagging or broken copy must not blank a companion that worked). It - asks whether the copy answered, not whether one is ready: `fallbackTransport` marks the responses - it builds with the internal `X-Audiosilo-Mirror` header (`localAnswerHeader`; stripped from remote - answers, never sent to any client) and `getJSON` turns such a 404 into `errLocalNotFound`, which - wraps `ErrNotFound` (so `errors.Is(err, ErrNotFound)` holds everywhere else). The remote service's - "no match" (no ready copy, or a request the copy failed) replaces the row as in remote mode. - `Ping` is unchanged: its `/healthz` goes through the same client, so the copy answers it once - ready and the remote service until then (parity by construction; cached a minute as before). The - runner is `internal/metamirror.Mirror` (`New(dir, Options{Enabled, Now, FreeBytes, Logger, - SiteURL, Repo, Release})` - it builds the release client itself, adding - `release.WithBaseSize` - then `Run`, `CheckNow`, `Wake`, `Current`, `Ready`, `Handler`, `Status`, - `Close`; `Remove(dataDir)`): `/meta-mirror/` (0700; not in backups, which hold the - database only) holds exactly one `meta-.sqlite` (0600) and `state.json` (`jsonfile`: tag, - built_at, schema_version, size_bytes, downloaded_at - the one record of the copy's facts, - Status reads them there -, etag, checked_at, last_error, and published_at + sha256 as on-disk - diagnostics only). `New` is cheap (state + folder): `Run` opens the copy first (`query.Open`'s - integrity checks take seconds over 1.8 GB, which must not hold up the listener), so until then - the status says `opening` with `fallback` and lookups go remote (a finished download is - `opening` too while `query.Open` runs, before the swap: over a working copy that copy still - answers, so no `fallback`). Schedule: durable, at most once a - day from the `checked_at` on disk (a restart does not re-download), 30 s after start without a - copy, an hour after a failed check, and on `CheckNow` (which makes `next_check_at` now; a running - check shows none until it ends); never while `metadata.enabled` is off (a check in flight, its download - included, stops when it is turned off and records nothing: `checkWhileEnabled`, and a Check now it was - answering stays due; while off `Run` - looks again each minute, and the settings PATCH that turns metadata back on calls `Wake`, which - re-reads the schedule at once but makes nothing due: only a due check runs). The release-list request is conditional (the ETag is sent only while a - copy is held, and dropped after any failure following a 200, so a 304 can't hide a release that - failed). `update` is list then `install` (tag check, disk guard, download, open, swap) with one - ETag reset on its error. A tag that can't name a file is refused (`validTag`); the disk guard - needs max(4.5 x the gz size, the current copy) + 512 MiB free (`not enough disk space: need X, - have Y`, decimal units as the console's `formatBytes`); the download goes through `release.DownloadData` (temp `.meta-*.tmp` in the folder, - `release.WithBaseSize` = the current copy's size), is opened with `query.Open` (integrity checks; - a file that doesn't open is deleted), then swapped in atomically; the replaced copy is closed and - deleted after a 60 s grace (a release swapped back in within the grace reuses its name: that - install closes the retiring handle at once without deleting, and no retire deletes the path an - install is writing or the current copy's). Any failure keeps the current copy. At its start `Run` deletes temp - files, a half-written state and any copy but the current one, and forgets a copy that is missing, - changed size or doesn't open. A copy newer than `MaxSchemaVersion` still opens and answers (flagged - `schema_newer`); artifact schema changes must stay additive (CROSS-REPO §17). The launcher - (`pkg/launcher/metamirror.go`) builds it only in mirror mode with a metadata service (release - client: `release.DefaultRepo`, `User-Agent: AudioSilo/`; tests point it at a fake - GitHub through the unexported `Options.metaReleases`), builds it right after `api.New` and hands - it over with `API.SetMetaMirror` (which calls `meta.Service.SetMirror`) BEFORE the background - jobs start, and a server started in remote mode deletes the folder. - Covers are unchanged (their own hosts in both modes). Admin wire below (Server settings). - **Bundle** (`meta_bundle` capability): `?include=previous` adds `previous` - (`meta.PreviousWorkIDs` / `Service.Previous`: main-view works before this one - (an omnibus "1-3" only when it ends before this one starts), nearest first, max 5, failures left out) and `?spoilers=hide` gates the current - work by the CALLER's saved progress (`meta.ChapterAt` + `meta.HideSpoilers`, - mirroring the player's `meta-gating.ts`; previous works lose only - `recap_summary.ending`). Both work on a per-request copy - the cached envelope - is shared and immutable. `MetaWork` also carries `community_description` - (CC BY-SA, apart from `description`) and `attribution` (present iff the work's - COMMUNITY content has characters/recaps/recap_summary/community_description: - `carriesCommunityContent`; generated content, laid in after it by - `meta.Answer`, never claims the credit); `MetaRecording` - carries `chapter_count`. `Service.CachedWorkID` answers an identifier's work id - from the cache alone: memory, else its enrichment's stored row, fresh or stale - (never the upstream), so placement does not flip with a restart or an eviction. - **Owned entries (`local`)**: every rail entry, main view and each - `orderings[].works[]`, carries `local` `{library_id, path}` when the CALLER owns - that work. Resolved per request AFTER the cache, on a copy of the rails (never - stored: the cached envelope is shared): `api.localRails` fetches the caller's - candidates (`catalog.SeriesBooks` over their `UserScopes`, scope-filtered: books - whose series - the main one (`books.series_key`), or one their `more_series` names - (`seriesMembers`, over `book_more_series`, migration 0041) - folds to a - rail/ordering name by `match.SeriesKey`, the console's `seriesKey` - NFKD, case, - punctuation and spacing ignored; a book in several series is a candidate in each, - at its position there) and `meta.PlaceLocal` - places them: the current work's entry -> the requested book; else the book whose - work id `CachedWorkID` knows; else (orderings) the main view's book for that work; - else the series position == the entry's numeric position on a series named like that - view (a range entry "1-3" holds a book at that range; an entry at one number a book at it, - else a book whose range starts there; `parseSpan`). A book with a known work id is never placed by index; one book per entry; - ties: the requested book's library, then library sort order, then path. - **Next book** (`next_book` capability): `GET /libraries/{id}/next?path=` - (`handlers_next.go`, `authorizedScope` + `bookForPath` like `item`) answers - `{source, next?, book?, work?}`; `source` names the step that produced `next` - (or decided there is none): `community` (metadata on + matched + a rail: - `meta.NextRail` reads, off the shared envelope, the rail that decides: in the - book's own order (`railOrder`: the rail named, by `match.SeriesKey`, like its main - series or one of that rail's orderings, then like its more_series entries in list - order, then the rest as listed) the first with an entry after the current work - that doesn't step back decides, with its first such entry (`nextEntry`: on its - MAIN view, the smallest numeric position (a range "1-3" by its start) above the current - work's last (a range's end), ties in rail order but a single position before a range - starting there, passing over each entry that steps back, - `railStepsBack`: the entry's work at or before the current work's position on an - earlier-ranked rail, both numeric - after The Silver Chair the publication order - passes over The Horse and His Boy and The Magician's Nephew, earlier - chronologically, and decides with The Last Battle); `NextRail` returns the rail - and entry indexes, and `communityNext` reads that same entry off the placed rails - by them (`PlaceLocal` keeps every rail and entry at its index; the placed rail's - plain next entry could be a passed-over one); `localRails` runs once, only when a rail - decides; its entry placed -> next + book + work. A later rail's placed entry never answers - instead: an unplaced deciding entry rides along as `work` without `local` beside - the steps below, since failing to place (untagged, series named unlike the rail) - proves nothing; current work last on every rail (they can lag the library), - upstream error/unmatched/no rails/unnumbered -> no `work`), `series` - (`catalog.NextInSeries`: every series of `Book.AllSeries()` with a position, main - first; per series one query - (`laterSeriesMembers`) over its numbered members in scope - same library, books in - that series (any spelling: by its key, see Series spellings) by their main series or a more_series entry, at their - position IN THAT SERIES - those above the book's position (a range's end) first, then by - position, then a range's end (a single book before an omnibus starting with it), ties by path (unbounded: SQLite sorts every member first anyway): - read in that order, the first row above that doesn't step back is the next book, - and any row at all means numbered (the first at or below ends the walk); a UNION - ALL of an `idx_books_series_key` branch and a `book_more_series` branch (its key - index, then each book by rowid); the first series with a later book that doesn't step back - answers with the first such book (`stepsBack`: it sits at or before the current - book (its last position: a book inside an omnibus steps back) in an earlier-ranked series of the current book's, both numbered there; a - book that steps back is passed over, not the whole series); numbered books in - some series but none later (or every later one stepping back) in any -> - `{source:"series"}`), `folder` - (`library.NextSibling` over the parent's whole listing, `ListDir`, scope- and - ignore-filtered, annotated by `BooksByPaths`, which reads any number of paths in - chunks; the player's `findNextSibling`: names compared as its `localeCompare` - (numeric, base) does, by `x/text/collate`, and a bare folder only when nothing in - the folder, the current book included, is indexed), else `{source:"none"}`. A - failure to place the caller's books leaves the next entry unplaced (as `/meta` - degrades). `book` is the list shape; everything `next` names is in the caller's - scope. -- **Native deep-link association**: `GET /.well-known/apple-app-site-association` - and `/assetlinks.json` are served from `config.AppLinkConfig` (`app_links` in - YAML) and 404 when unset. They only enable auto-app-launch for domains the - shipped app build claims - self-hosted arbitrary domains fall back to the web - player + the custom-scheme "Open in app" button. -- **SQLite** runs with a single write connection (writers serialize) plus a - read-only reader pool, WAL mode. `store.DB.QueryContext`/`QueryRowContext` go to - the READER (`query_only`): any statement that writes, including `INSERT ... - RETURNING`, must use `ExecContext`, `WriteRowContext` or a `WithTx` transaction. - An in-memory DB has reader == writer and would hide a misroute, which is why the - test stores are temp files. -- **Pagination** is keyset/cursor-based (`catalog.ListBooks`); don't switch list - endpoints to OFFSET for large tables. -- **Path safety**: any filesystem access derived from user input goes through - `library.SafeJoin`, which rejects traversal outside the library root. -- **Path-addressed API**: content endpoints are `GET /libraries/{id}/{item, - chapters,cover,stream}?path=` and `{GET,PUT} .../progress?path=` etc. The path - is the handle (a query param, to avoid encoded-slash issues). `item`/`chapters`/ - `cover`/`meta`/`next` resolve `(library, path)` to a book in ONE place, `bookForPath`: - `GetBookByPath`, then the indexed folder book holding the path - (`GetBookHolding`: a part, a disc folder of a joined book), then indexing on - demand (`Scanner.IndexPathWithin`) if the scan hasn't reached it. A book found - above the requested path must be in the caller's scope too (`library.ErrNotAllowed`, - answered as the out-of-scope 403 `no access to this path`, nothing read or - indexed for that caller); `stream` serves an - audio file path directly, or transcodes it with `?transcode=1` (see below). The - `/fs` view lists **audio files and directories only** (non-audio like `.jpg`/ - `.nfo` are filtered in `BrowseFS` so a click is always playable) and annotates - book entries with metadata (`is_book` + title/author/…); the client acts on the - entry's `path`. -- **Recognized audio** is `metadata.AudioExtensions` (`IsAudio`), including `.mp4` - (AAC-in-MP4 audiobooks); `media` serves it as `audio/mp4`. -- **Transcoding**: `GET /libraries/{id}/stream?path=…&transcode=1` pipes the file - through ffmpeg to MP3 (`media.Transcode`) for codecs browsers can't decode; - `&t=` starts mid-file (transcoded output isn't byte-seekable, so seeks - re-request). Direct serving + Range stays the default. The scanner records each - book's audio `codec` (ffprobe `codec_name`, `0008` migration); `item`/`chapters` - expose `direct_playable` (via `media.DirectPlayable`) so a client knows when to - transcode. Gated by the `--ffmpeg` flag (the `transcode` capability reflects it). -- **Sharing & access (shares)**: access is via `shares` = named sets of path - rules. `catalog.UserScope`/`UserScopes` build a `Scope` per library - (`AllowAll` or specific `Paths`); `Scope.Allows` gates item endpoints, - `Scope.VisibleInBrowse` filters `/fs` to a navigable subtree, and - `pathFilterSQL` scopes `ListBooks`/`Search` and the player's browse aggregates - (`People`/`Series` given a `*Scope`, which also pins them to the scope's library; - nil = the admin's unscoped view). Every content handler authorizes - the path against the caller's scope (`authorizedPath`). Admins are `AllowAll`. - Whole-library access is sugar (`GrantWholeLibrary` → a `""`-rule share). -- **Move-tracking**: the scanner fingerprints files; when a path vanishes and a - new path with a matching fingerprint appears, `Scanner.detectMoves` migrates - durable state old→new (`catalog.MoveDurableState`). Re-tagging keeps state via - the path key; moving keeps it via the fingerprint. The per-user path-keyed tables are listed - ONCE, in `carryListeningState` (`catalog/listening.go`; add a table -> add a line), which both - a move (offset 0) and a join (`JoinDurableState`) use: positions shifted onto the new - timeline, a listener's progress already at the new path merged by a strategy the caller - passes (`progressMerge`), a favourite landing once, so a collision never fails a move. A - **move** takes the newest save whole (`mergeNewest`: `updated_at`, then version; a row - already at the new path is a removed book's, so its position or finish must not win by - being further on); a **join** keeps the furthest (`mergeFurthest`: the rows are parts of one - book; finished over not, the newer save's stamp, the earlier start). Both take a version - above both rows. A favourite on a navigation folder (author, series) has no book to move: - `detectMoves` re-keys it (`catalog.MoveFolderFavourites`) to the folder its moved books - went to (`renamedFolders`), only when every move out of it agrees and it is gone from disk - by exact name (so a case-only rename, of it or a folder above it, counts). -- **Auto book/folder detection**: there is **no per-library layout**. The model - (`booksInDir` in `library/scanner.go`) matches the dominant "folder per book" - convention (and Audiobookshelf): **a directory that directly contains audio is - ONE book**, with all those files as its tracks/chapters - whether it holds a - single m4b or fifty distinctly-named mp3 chapters (do NOT split a folder's files - into separate books by filename; that produced one phantom book per chapter). - The only per-file case is the **library root** (loose files there are individual - single-file books - the old "flat"). A folder of loose single-file books - (`books_in_folder`) is expressed with a **per-folder override**: - `folder_overrides(library_id, path, mode)`, `mode ∈ {book, collection}` - - `collection` = one book per file, `book` = force folder-is-one-book. Overrides - are durable, path-keyed config (no FK to the rebuildable index, like - progress/bookmarks). `PUT/DELETE /admin/libraries/{id}/folder-override?path=` - sets/clears it and rescans; the admin console's per-library **Folder detection** dialog - drives it. `GET /fs` annotates each entry's effective `override`. - **Joined books (`library/joined.go`)**: `book` joins only a **disc set** (`discSets`): a - folder below the root with no audio of its own whose every folder holding audio beneath - it is a disc folder (`isDiscFolder`: cd/disc/disk + number) **directly** in it, at least - two (`minDiscs`; one disc already reads as one book). A CD rip (`Book/CD1`, `Book/CD2`, - ...) becomes ONE book of its discs' audio (hidden/ignore rules applied), with no books for - the discs. Any other folder keeps `book`'s old meaning (its own files; without any, a - no-op): overrides the old detection dialog set on author/series folders must never merge - a series (or its listeners' progress) on upgrade, and nothing is auto-detected - (re-shaping existing books would orphan progress). Order: the disc folders by - `discOrder` (natural: CD2 before CD10, "CD 1" = "CD1", ties by path), each disc's files - in exactly the order a book of that disc alone has them (byte-wise name order, as - `audioEntries`/the walk read them; NOT natural), since the state carry-over maps a disc - position as offset + position. Files and chapters keep real paths in the discs - (`file_path`, a chapter title led by its disc: "CD2 - 01"); the cover is the folder's own - image, then the first disc's; `IndexPath` resolves the folder, a disc folder or a file in - it to the joined book, and so does `bookForPath` from the index first (`GetBookHolding`), - so the disc paths shipped clients still hold don't re-walk and re-probe every disc on - each request. Reaching the joined book through a disc path takes a grant on the book: a - share made before the join on one disc folder gets the out-of-scope 403 for it (as for - the book's own path) and never triggers the re-read, though it still streams the disc's - real files (scoped on the file path). Discovery and `IndexPath` decide a join in one place (`joinRoot`: - the folder or its parent, with `book` and a disc set), and the discovery walk - (`audioDirs`) reads each folder's audio files once, handing them to - `booksInDir`/`joinedBook`. The scan where it takes effect runs `carryJoinedState` before - the prune: `catalog.JoinDurableState` moves the disc books' listening state onto the - joined timeline (offset = earlier discs' lengths; furthest progress per listener wins; - bookmarks/notes/history/sessions offset; favourites once) and copies their - edits/cover/enrichment (earliest disc wins per field, chapter renames re-keyed). A disc's - listening state is carried only when its offset is **known** (`joinParts`): the first - disc's always (0), a later disc's only when every earlier disc's length is (its files', - else its indexed length). With ffprobe off or failing, a later disc is `Unplaced`: its - listening state stays on its own path (path-keyed, back if the join is undone) rather - than landing in disc 1, its config is still copied, and its `joined` event carries code - `length_unknown` (the console: "progress stayed with the disc: length unknown"). Each - disc is logged `joined`, not `removed`, and the joined book is a reshape, not new - content: not counted in `Added`/`AddedTitles` (`joinsIndexed`), so no "books added" - notification. Removing the override splits the discs out again; the joined state stays - on the folder's path. That un-join is a reshape too (`Scanner.splitFrom`, the - `joinsIndexed` counterpart: books in the disc folders of a folder whose stored book was - joined, `isJoined`, and is no book now): the discs are not counted in - `Added`/`AddedTitles`, and the joined book is logged `split`, not `removed` (nor counted - in `Removed`). A folder book with audio of its own that goes is still `removed`. `GET /fs` - marks a disc set whose discs are indexed (`split_discs`, from `books.split_parent`) for an - **admin** caller only (`annotateWithBooks`; `omitempty`), so the player's listing is - unchanged; the console offers "Always one book" on a folder without audio only there or - on a joined book. -- **Metadata overrides (admin redesign Phase 2a, `catalog/overrides.go`)**: an admin's - edits never touch files. A `books` row holds the **effective** values - what the scan - found (kept in `books.scanned`, JSON field -> value, stamped with the upsert's - `indexed_at`: a snapshot whose stamp isn't the row's was left by an older server, so - `loadLayers` reads the scan's values off the row instead), then an Audiobookshelf - `metadata.json`'s values (`books.abs_file`, below), then `book_enrichment` - (asin/isbn), then `book_overrides`. `bookLayers.resolve` is the ONE statement of that - precedence (and of each field's source); `refreshEffective` writes its values to the - row (plus chapter titles from `chapters.scanned_title` + `chapter_overrides`, and FTS) - and the book page shows it as provenance, so the two can't disagree. `UpsertBook` runs - `refreshEffective` in the scan's own transaction, which is what makes an edit a lock (a - rescan rewrites the scanned values and re-applies the edit before anything can read - the row) and why there is no separate post-scan enrichment pass any more. - `SetEnrichment` and `EditBook`/`EditBooks` call it too. Players, search, `/fs` and - export read the row, so they see edits with no join. The player's book JSON carries - `published` on every book and `description` only on `GET /libraries/{id}/item` (api's - `itemBook` adds it; `catalog.Book` keeps it `json:"-"` so list, search and recent pages - stay small); `has_cover` and per-file codec are admin-only (`json:"-"`). Every player book (list, - search and item alike, all `omitempty`) also carries `subtitle`, `publisher`, `language`, - `abridged` (a bool, absent while unknown) and `genres` / `tags` (lists). Validation: `normalizeOverride`; sources: a scanned value is `path` when - it equals what `DeriveFromPath` (or, path-first, `FromPathLayout`) yields, else `tag`; one an ABS - file gives is `abs`; an override is `edited` or - `community`; an enrichment-attached ASIN/ISBN reads as `community`. Revert = delete the - override + `refreshEffective` (restores the scanned value, the ABS file's where it gives one; no - reindex, no disk). - **Audiobookshelf metadata.json** (`metadata/absfile.go`, `catalog/absfile.go`, migration `0046`): - ABS keeps a `metadata.json` in a book's folder ("Store metadata with item"), its record of the book - with its matches and its users' corrections, often holding the ASIN/ISBN the tags don't. A folder - book's file (a regular file named exactly that, not ignored: an ignore rule `metadata.json` turns it - off for a library; never for a file book, nor a loose file at the root) is a layer OVER the scanned - values, tags and path alike, path-first libraries included (ABS's own default ranks it above the - tags): `bookLayers.overABS`, in `scannedFields` before `titlePosition`. `metadata.ParseABSFile` is - lenient (a wrongly typed value costs that value; <= 4 MiB): title, subtitle, authors and - narrators joined with ", " (strings or `{name}`), series `"Name #seq"` (the first is the main series - with its position, the rest `more_series`; a series without a position keeps the tag's only while - it is the same series, by `seriesKey`), `publishedDate` else `publishedYear` as `published`, the - HTML description as plain text, publisher, language, `abridged` (only `true` says anything: ABS - writes false for every book nobody marked), genres and tags (lists of strings, a non-string item - dropped; `CleanLabels`), asin, isbn; chapters are not read. An empty value says nothing (never clears a tag's). `absFields` stores each as an edit of its - field would (`CleanOverride`; a refused value left out), so an ABS ASIN makes a book matched (Health, - `/meta`, the community chapter check, a refresh run). `books.abs_file` holds those fields (JSON, - `''` for none), `abs_sig` the file's signature (`metadata.ABSSignature`: size + mtime ns, `''` for - none) and `abs_indexed_at` the `indexed_at` it was read for (NULL before `0046`), narrow columns - first so `Signatures` reads no JSON; `UpsertBook` writes all three (`absColumns`; nothing for a - file book). Discovery records each folder's file signature from the walk itself (`audioDirs`' - `absFiles`, the walk's own entry, no extra directory read; `IndexPath`, with no walk, stats it: `absSigAt`), and `enrich` reads - the file only when there is one (`Scanner.readABS`). An unchanged book whose signature differs, - or that has a file not read for this index (`abs_indexed_at` NULL, or another index's: an older - server re-indexed it), is read without a re-index: `Signature.ABSSig`/`ABSUnchecked`, - `Catalog.SetABSFiles` in batches of 250 beside the path backfill (`eachPathBatch`, shared with - `SetPathReading`); a book with no file and none before is left unwritten, so the first scan - after the upgrade touches only books that have one. `setABSFile` re-resolves a book unless the - reading equals the one recorded for this very index (an older server's re-index resolved the - row without it). `metadata.ReadABSFile` reads a regular file only, never through a link (one - swapped in after the walk included: `os.SameFile` on the opened file). A file that can't be used - (not JSON, too large, not a regular file: `metadata.ErrABSFileInvalid`) keeps its signature (read - again once it changes); one that can't be read right now (gone since the walk, a flaky mount) is - left as it was by the backfill and recorded as none by a re-index, so the next scan reads it - again (`readABS`' `read`); both log a warning. Known gap: a position read from an ABS title (`titlePosition`) shows as `tag`. - **Bridge attachments** (`metadata/attachments.go`, `library/attachments.go`, `catalog/attachments.go`, - migrations `0048`, `0050`; BRIDGE-PLAN.md Phases 1a and 1c): the Bridge writes a book's generated `characters.json`, - `recaps.json`, `description.json` and `bridge.json` (provenance) into `.audiosilo/` in a folder book's own folder, or beside - a single-file book with its stem before each name (`Book.characters.json`). They are NOT a metadata - layer (nothing in the effective row reads them): the scan stores them checked (audiosilo-meta's - `model.Characters` / `model.Recaps` / `model.Description`, bounded, a regular file only, never through a link) in - `book_attachments`, and `/meta` serves them (below). A description (`metadata.ParseDescription`) is - its text trimmed, 200-1500 code points (meta's `description.schema.json` bounds, - `TestDescriptionBoundsMatchTheSchema` reads them from audiosilo-meta's embedded `SchemaFS`, imported - as `metaschema`), no control character but a line break; a blank text is none, anything else fails - the file. A signature lists only the names a book has, in `AttachmentNames` order, and adding the - name kept the other three in order, so a book without one keeps its stored signature byte for byte. - bridge.json's `description.generated_at` is read as `AttachmentProvenance.DescriptionAt` (the rest of - that object is the Bridge's; an object of another shape costs only that time, never the file). The discovery walk lists each `.audiosilo` folder - ONCE (never walks into it; no audio lives there), and a book's signature (`books.attachments_sig`: - name:size:mtime_ns of its own known names) is compared like `abs_sig`: an unchanged book whose - attachments are new, changed, gone or not read for this index is read without a re-index - (`SetAttachments`, compare-and-set against the state the scan began with, so a scan never lays an - older reading over the Bridge's refresh since). A file that can't be used keeps its signature (read - again once it changes); a folder or file that can't be read right now keeps what is stored (`Unread`), - never taking a book's generated content away. `Scanner.RefreshAttachments` re-reads one indexed - book's attachments, never its audio (the Bridge's `POST /bridge/rescan` mode `attachments`). The - player's `Book.generated` (list, search, item, recent, the next book) is true when the signature names - a characters, recaps or description file (`sigParts`, an allow-list: bridge.json and any name a later - server adds are none; `Book.GeneratedParts`, off the wire, says which; the narrow column, no join), so the player asks `/meta` - for a book with no ASIN/ISBN too. Derived from the files: rebuildable, never carried by a move. - **Series ranges** (an omnibus of books 1-3): the `series_index` field may hold a range, - canonical `"1-3"` (input `1 - 3` and the en dash too; `metadata.ParseSeriesRange`: 0 < start < - end <= 100000, a span of at most `metadata.MaxSeriesSpan` = 100, since the slots a range covers - are counted one by one); `refreshEffective` writes its start to `series_index` (what every - client reads) and its end to `series_index_end` (`ParseSeriesPosition`; `FormatSeriesRange` the - stored text). A `more_series` entry carries one as `position_end` (`SeriesRef.PositionEnd`, - omitempty; `book_more_series.position_end`). Player `Book` JSON: `series_index_end` and - `series_list[].position_end`, both omitempty; the admin book row `series_index_end`. `seriesSwap` - carries a range both ways; the series aggregate's `positions` hold every whole number inside a - range (`heldPositions`), so a player marking gaps sees none inside an owned omnibus. - An admin edit that makes the main series a series its `more_series` lists (any spelling, `match.SameSeries`) - - by setting `series`, or by reverting it when the series the revert leaves - (`newMainSeries`: `loadLayers` + `resolve` without the series override) is listed - - without setting or reverting `more_series` itself, swaps them (`catalog.seriesSwap`, - per book in `editTx`, from the book's effective row): the old main series takes the - entry's place in the list and `series_index` becomes the entry's position (unless the - edit names `series_index`), written as the edit's own overrides; so reverting a swap - swaps back. A derived value equal to what its field resolves to without an override - (`bookLayers.unedited`: the scan's or folder layout's value; layers loaded only when a - swap happens; a position only when the new main series is the series it numbers, by - any case) is written as a revert of that field instead (`seriesSwap` returns sets - and reverts, disjoint, never a field the edit names), so swapping back to a - path-derived Sherlock Holmes #5 leaves `series_index` read off the path (unlocked, - source `path`), while a list that was an edit before the swap stays one, and a - listed Other #5 made main on that path-numbered #5 is still an edited 5. An old main - series that can't be listed (name too long, position out of range) refuses the edit: - `invalid(series)`, a 400, failing a bulk edit whole (`EditBooks` names the refusing - book's path in the reason). A community edit never swaps: `matchrun.planSeries` is - the one statement of how a match lays out series. The console drafts the same swap before saving (`book-model.ts` - `commitField`), so its save sends the values and the server's swap doesn't fire; - `saveRequest` sends the saved `more_series` when the admin took the drafted swap back - (only for a swap `seriesSwap` can make, so an unlistable old main still gets the 400); - as that save names `more_series`, a swap the console drafts writes its values as - edits even where one equals the file's (the values shown are the same), - and the undo of a series revert that swapped (`undoRevertRequest`) sends the list and - position it had. `MoveDurableState` carries overrides and custom covers as one set: when the moved book - has any, the new path's own rows in all three tables are dropped first; it moves - them (with enrichment) in a transaction of their own, so a failure carrying the per-user - state can't strand them. `detectMoves` doesn't pair a folder reclassified as a collection (or - back) with its own first part, nor a joined book renamed away from its override with its first disc (`reclassified`). - **Metadata source** (`libraries.metadata_source`, `0032`; `catalog.MetadataFromTags`, the - default, or `MetadataFromPath`; admin-only on the wire as `metadata_source`, PATCH 400 - `invalid_metadata_source`): a path-first library resolves the scanned layer with the folder - layout over the tags (`bookLayers.scannedFields`: `metadata.FromPathLayout`, where the top - folder is the AUTHOR and disc/track folders are parts, unlike `DeriveFromPath`'s two nearest folders; a layout value goes - over a tag's wherever it says anything and replaces the baseline's own reading even with - nothing; the title always keeps one, and a tag title that IS the folder's name, number and all ("13 Reasons Why"), stays whole with no position read from it; the position goes with the series: the layout's own, else - the scanned one only while the series it numbers stays). It is a resolve rule, not a scan one: the snapshot is unchanged, - and `UpdateLibrary` re-resolves the library's books (`refreshLibrary`) in the edit's own - transaction, no rescan (about 1 s per 5,000 books). - `has_cover` holds whenever there is a sibling cover (`UpsertBook` enforces it) and is - NULL until checked; the scanner backfills unchanged pre-0016 rows in one transaction - with a tag read (`media.EmbeddedCover`, no ffprobe). A `scanned` snapshot carries `@rev` (`scannedRevKey`), the path baseline's revision: one - without it (any older server's) was read when a lone folder above the book was its series. - `Signatures` classifies such a book one folder deep (`pathCheckExpr`): settled by the snapshot - alone (a tag's series and no author, missing or 0016's "": the folder is the author; the folder - as series and author: the series was the path's), or only by its tags (the folder as series, - another author or none: "Discworld/Mort" may be tagged Discworld; and any stale or blank - snapshot, whatever revision it names). The scan reads the latter's primary file once - (`readNames`, shared with `enrich`; not while ffprobe newly fails), and `Catalog.SetPathReading` - rewrites both kinds' snapshots (skipping one re-indexed meanwhile, keeping an older server's - chapter titles) and re-resolves them in batches of 250, flushed every 500 during the scan and - on a Stop; no re-index. Until then `bookLayers.pathBaseline` resolves a snapshot against the - baseline it was read with, so a path-first library still drops the folder as a series. - **Labels** (`catalog/labels.go`, migration `0049`): `genres` and `tags` are override fields whose - value is a canonical JSON list (`""` for none; `normalizeLabels`: trimmed, spacing collapsed, each - label once by `labelKey` (lowercased, spacing aside) keeping the first spelling, <= 64 characters, - <= 50 labels; an edit over a bound is a 400, a file's list is cleaned leniently instead, - `CleanLabels`, which every file source goes through). The value is `encodeCanonical`'s JSON (no - HTML escaping), as `more_series`' is. Genres come from the genre tag (`metadata.SplitGenres`: split - on `;` and NUL, never `,` (a store category holds one), blanks and placeholders such as "Audiobook", "Speech", "Spoken Word" or - "Other" dropped, any case; the rest left to `CleanLabels`; dhowden's then ffprobe's, ffprobe's when - it gives any, in the `scanned` snapshot) and the ABS file; tags only from the ABS file and edits. - `subtitle`, `publisher`, `language` (<= 64 characters) and `abridged` (`"true"`/`"false"`/`""`, - stored 1/0/NULL) are plain override fields from the ABS file, a match (`matchrun.Propose`: the work's - subtitle and language, the recording's publisher; community genres are not offered, a match - candidate carries none) or an edit. A source's abridged false says nothing (`AbridgedClaim`: ABS and - the community write false for every book nobody marked), so only an edit says unabridged. - `refreshEffective` writes the lists to `books.genres`/`tags`, to `book_labels` by key - (`writeLabels`, beside `keyOtherSeries`) and to the FTS `genres`/`tags` columns (a label a line), so - a search finds a book by a label. `labelFilters` (in the library, `book_labels.library_id`; any - library for an admin list across them and for the library facet) is the player's `GET - /libraries/{id}/books?genre=&tag=` (one value each, any case; both narrow) and the admin - `BookFilter.Genre`/`Tag` (`?genre=`/`?tag=`); `BookFacets` counts `genres`/`tags` (the 100 most - common by key, then the one the list is filtered by when it is rarer, each named by the spelling - most of its books use, `seriesName`; a semi-join over `book_labels`). A bulk edit (and the book PATCH) takes `add` / `remove` (`{"tags": ["Owned"]}`: - field -> labels) besides `set` / `revert`: `labelSets` merges them into each book's own EFFECTIVE - list and writes the result as an override of the field only where it changes the list; a book the - edit changes nothing on (adding a tag it has) is left as it is, its row not even rebuilt (`editTx`). - A field can't be set or reverted and added to or removed from at once, nor one label added and - removed (`normalizeLabelEdits`). - Books indexed before `0049` get their genres from the next scan (`genres_checked`, the primary - file read once beside 0037's date read, `Catalog.SetGenres` writing the snapshot and re-resolving, - flushed with the path readings) and the ABS fields from its re-read (`abs_indexed_at` NULL), no - re-index. Path-keyed like every override, so moves and joins carry them unchanged. -- **Series spellings (`catalog/moreseries.go`)**: a series is every name with one - `seriesKey` (`match.SeriesKey`, the rails' key: case, accents, punctuation and spacing - aside; a name with no letter or number is its own key), so "He Who Fights With Monsters", - "...with..." and "...WIth..." are one series and nothing fuzzier merges ("The Expanse" and - "Expanse" stay two). `catalog.Series` groups by it (counts, positions, lengths and authors - merged; `SeriesList` and `normalizeMoreSeries` keep a series once however a list spells it again) and names the group `seriesName`: the spelling its - matched books use most, else the most books', ties alphabetical. **The key is stored** - (migration 0041): `refreshEffective` writes `books.series_key` (`seriesKey`, '' for no - series; index `idx_books_series_key (library_id, series_key, series, series_index)`, which - replaced `idx_books_series`: spellings adjacent, so Health's `series_spellings` is two range - seeks per book) and the book's - other series to `book_more_series` (`keyOtherSeries`: `SeriesList` less the main series, one - row per key with its position; PK `(book_id, series_key)`, index `(series_key, library_id, - position)`, cascades with the book). Both are written by Go, never computed in SQL (the key - needs NFKD; the catalogue's SQL functions stay out of the schema), and every write of - `books.series` ends in `refreshEffective` (`UpsertBook` runs it in the scan's own transaction), - so every path that changes a series keys it. Every series - filter asks by key: `seriesFilter` (an `id IN` over `idx_books_series_key` in every library, - with memberships a UNION ALL of the list rows too; the caller's library term narrows them), - `seriesOrderKeys` (the key as a literal), `numberedSeriesMembers` (the next book), - `SeriesBooks`/`seriesMembers` (the rails) and `positionIn` (Go, by key), so the SQL and a - keyset cursor agree; each is an index seek (`TestNextInSeriesPlan`; under a millisecond for a - 25-book series in 50,000 books, `TestSeriesKeyScale`). Rows indexed before 0041 are keyed by - `Catalog.KeySeries`, which the launcher runs synchronously before the listener whenever the - stored keys are missing or out of date (server_state `series_keys` holds `seriesKeysVersion`: - the key rules' number, bumped when `seriesKey`/`match.SeriesKey` changes, plus Go's and - x/text's Unicode versions, so a toolchain upgrade re-keys); about a second per 50,000 books, and - a failure stops the start, as a migration's does (the next start tries again): it first puts every stored `more_series` override through - `normalizeMoreSeries` (a list written before #123 could name one series twice) and rebuilds - those books, then keys every book in one transaction. The JSON - is unchanged: an older player sees fewer, fuller series. **A position from the title**: - `bookLayers.scannedFields` gives a book with a series and no position the one its title gives - (`titlePosition`, `metadata.PositionInTitle`), read as the title's source (path when the title is - the path's, else tag); an edit or a tag's own position wins. Rows indexed before the rule get it - from `Catalog.PlaceByTitles`, which the launcher runs in the background at start, once per database - (server_state `title_positions`): the books with a series, no position and a digit in the row's, the - snapshot's or the path's title (a match may have replaced the row's) whose resolve gives one, - re-resolved by `refreshEffective` in batches of 250. -- **Admin catalog API** (`api/handlers_catalog.go`, admin-only, transport-only): - `GET /admin/books` (keyset over the named orderings in `catalog.adminSorts`, sorted and - paged on ids before the per-row columns are computed; the cursor names its ordering; - filters in `catalog.BookFilter`, unparseable ones 400; `media.DirectPlayableSQL` is - `DirectPlayable` in SQL) + `/admin/books/facets` (each dimension counted without its - own filter, the unfiltered yes/no ones in one pass); - `POST /admin/books/bulk` (one edit over <= 1000 books, all or nothing); - `GET /admin/authors|narrators` (each person a credit names, `catalog.creditNames` - over `names.Split`, so a co-credit counts for both; `merge_suggestions` stay over - WHOLE credits, from `personKey`, keyed by `match.Fold`, which keeps every script's - letters, with `other_books` = the books a merge rewrites) and - `/admin/series` (`catalog.People`/`Series` with a nil scope; the player's - `/libraries/{id}/authors|narrators|series` take the same aggregates within the - caller's scope, without `merge_suggestions`); `GET`/`PATCH /admin/libraries/{id}/book?path=` (book - page: per-field provenance, chapters, files, listeners, shares, folder override, and - `match_query`, the match dialog's search box prefill set by the api: `meta.SearchPrefill`, the - title + author, except that swapped tags (the title is the author folder, or the cleaned author - is the path's title; never a tag that matches its own path fact, "Dune/Dune") or a junk title/author - (`metadata.NamesNothing`, "Unknown", "Various Artists") - take the folders' facts, `metadata.FromPathLayout`); - `GET /admin/libraries/{id}/book/match?path=` (`meta.Service.Candidates`: metaserve's - STRUCTURED match `works/match`, then up to 6 works expanded, uncached, bounded by - `workSem` via `fetchWork`. The match gets the book's facts separately - (`matchParams`): the path's LAYOUT (`metadata.ReadPathLayout`: disc/track folders dropped, - top folder = author, holding folder = series, leaf = a title guess sent RAW - - metaserve is the one place that reads `SW06 - `/`Sharpe - 08 - `/`02. ` - numbering and takes the position from it) then the tagged title, path + tag - authors, the series folder AND the tagged series when they differ (folder first; - metaserve judges each, the better counts), the tagged position (only when no - DIFFERENT series folder goes up: one position applies to every series guess and - replaces the leaf's numbering), runtime, `?q=`, and the asin/isbn (typed; - the book's only when nothing was typed), which metaserve looks up itself; candidates carry metaserve's - `score` and `reasons`. A typed `?asin=`/`?isbn=` with no text is a plain `lookup`. - Fallback, FROZEN until the production metaserve serves `works/match` (then delete - it): when the route is missing (404 through `works/{id}`, 405, or a 200 without - `results`) `searchHits` runs the old `lookup` + `works/search` unchanged (typed - text, else `CleanTitle` + author) scored by the tag-only `scoreCandidate`, and - the missing route is remembered for 15 min (`matchUnsupportedUntil`); a 503/5xx - from works/match is an outage (502), never a fallback; identifiers normalized for - the exact lookup; metadata off -> 404 `metadata_off`). Each recording keeps its - ASINs' marketplaces (`asin_refs`) and `asins` is ordered by `metadata.region` - (`orderASINs`: the preferred store's, then `us`, then the rest; metaserve lists - them by region, so without it `au` beat `uk`), `asin_region` names the first's; - `meta.DefaultRecording` (identifier's, else closest runtime, the preferred - marketplace breaking a tie) sets each candidate's `default_recording_id`, which - the dialog starts from (the console has no copy of the rule). **Bulk matching** - (`internal/matchrun`, `handlers_match_runs.go`, migration `0033`): - `GET/POST /admin/match-runs`, `GET /admin/match-runs/{id}[/items]`, - `POST /admin/match-runs/{id}/apply|cancel`. One run at a time (`matchrun.Runner`, - its own goroutine under `baseCtx`, NOT the scan job queue: it waits on the - network, and `library` must not import `meta`); 2 workers, `Limit: 2` works - expanded per book, 5 failures in a row stop it (`metadata_unavailable`), metadata - off stops it (an apply too: back to ready, as a cancel; every endpoint, cancel - included, 404s `metadata_off` then). A run only records (`match_run_items`: outcome auto/review/none/error, - the best candidate's `catalog.MatchProposal`); apply writes community overrides - via `EditBook` and the cover via `saveMatchCover` (the dialog's fetch + keep - checks) per `matchrun.Plan(scope)` (ids|fill|overwrite; never an `edited` field, - never a position beside another series, a cover only for a book with none), which - also computes each item's `changes` for the review, so the two can't disagree. `Plan` - returns `Writes` (set, revert, cover): overwrite REPLACES a book's series with the - community's for its recording (the series rules), reverting a `series`/`series_index`/ - `more_series` value the community wrote earlier that it no longer offers (a series it - deleted or merged, an edition or translation the rules leave out) back to the files' - (a position only beside the files' series, other series only when none are offered); - fill and ids never revert; an admin's own series stay, and a community edit's revert - deletes only community rows (`editTx`), so an edit made since the run looked survives - a revert (a set still overwrites it, as the match dialog's ticked edited field must). - `Writes.after` is the one rule for a field's value once written (both `planSeries` - and the review use it). A refresh records only books a scope would change (as a - repick records only books with a new ASIN), so a clean library leaves no items. A refresh - proposal names the identifier it was found by (`found_by`, "asin:..."/"isbn:..."), and - applying it is a no-op for a book that no longer has it (matched again since). - Likewise a match run's book that has an ASIN or ISBN now (`matchedSince`: matched by hand - from the review or anywhere, or a rescan found one) has no changes, reads `matched_since` - in the review and is skipped with detail `matched_since`, under any scope; a match whose - fields went in before a stop cut its cover short stays to apply, detail `cover_pending`, its - ASIN the run's own rather than one matched since, so the next apply takes the cover. The review's - items page best match first (`score DESC, id`), the cursor still an item id whose score - the next page reads on from. - Each scope's change carries `series` {from, to} (main first, position 0 = none) when it - changes the book's series, which the review shows as "old -> new". A retired work slug - is a redirect the client follows, so the names written are the merged record's. - A community position that is a range ("1-3") is written as one (`series_index` "1-3", a - `more_series` entry's `position_end`); one the field refuses (a span over 100) is none. - Confident = score >= 90, >= 10 ahead, with an ASIN/ISBN, and the recording within 10% of - the book's length when both are known (`meta.PlausibleRuntimeOff`, `lengthFits`). A - repick only touches an ASIN whose override is `source=community`. A **refresh** run - (`mode: "refresh"`, `catalog.MatchedBooks`: every book with an ASIN/ISBN) looks each - book up by its own identifier alone (the ISBN when the ASIN finds nothing) and takes - that record as confident, unless its recording is over 10% off the book's length - (`lengthFits`: then it is held for review) (`identifier_unknown` = no match, never a - fuzzy work); it fills or - overwrites the details but never writes an identifier (`planFor` drops them, the - proposal carries none), so `matchrun.ScopesFor` (the one statement of which scopes a - mode takes) allows only fill/overwrite (else `ErrScope`, 400); audited - `book.match_refresh`. `InterruptMatchRuns` (launcher, at - start) settles runs a stopped server left: matching -> interrupted, applying -> - ready. Newest 10 runs kept. `DELETE /admin/community-matches[?library_id=]` (`matchrun.Runner.Clear`, - 409 `match_run_busy` while a run works; works with metadata off) undoes the community matches so - books can be matched from fresh: `catalog.ClearCommunityMatches` drops the `source=community` - overrides and covers, rebuilds each book (`refreshEffective`, `refreshCoverArt`) and drops the - runs (a library's clear: its runs, and its items in runs over every library), one transaction; - edits, uploads, tags and `book_enrichment` stay. Audited `book.match_run|asin_repick|match_refresh|match_apply|match_stop|match_clear`; - `POST /admin/books/works` (`{books:[{library_id,path}]}`, <= 100, `catalog.BooksByRefs`) answers - `{"works":[{library_id,path,work_id,failed}]}` in request order: each book's community work id - (`meta.Service.WorkIDs`: per distinct normalized identifier, first the cached enrichment's - work id (a cached "no match" too), so a book a player opened costs the card no lookup, then its - own `"l:"` key space (Enrich's TTLs, quota `maxLookupEntries`), which only `fetchLookup` writes - (Enrich's compose looks up fresh and records nothing there); only misses go upstream, - through `sharedLookup`: bounded across console requests by `lookupSem`, concurrent misses of one - identifier sharing one flight whose waiters can each leave without failing the others, the batch - under `composeBudget`, and a failure from the caller's, the flight's or the budget's context never - cached. One-way: Enrich never reads `"l:"` nor waits on `lookupSem`, so a console batch can't time - a player's `/meta` out or age its enrichment); `""` = no identifier, no match or a failed lookup, - and per book `failed` says its own lookup failed or ran out of time, so asking again may resolve it - (never for a clean miss); metadata off -> 404 - `metadata_off`. The Series cards place an owned book on a community rail by this work id first and - fall back to `series_index`, then to an unheld entry its title is (`sameTitle`: numbering prefix - and a trailing bracketed part ignored) (display only: no book is changed). A card's rail - (`pickRail`) is the one named like the series (or one of its `orderings`), then one named like - it give or take an article or a trailing "Series", then the rail holding >= 2 and more than half - of the series' resolved works (never the rail named like the series' name before a colon or a dash: the parent of a - sub-series; ties to the smaller rail), else none: no gaps rather than another series' (the rail - comes from one matched book's `/meta`, so a work in more than `maxSeriesRails` real series can - still lack one); - `PUT`/`DELETE /admin/libraries/{id}/cover?path=` (custom cover in the DB; - `catalog.SetCover` enforces 5 MiB, sniffed JPEG/PNG/WebP and an indexed book), which - `GET /libraries/{id}/cover` serves first (by the requested path, then by the book a - part path resolves to; only while a book is indexed there), behind the caller's - scope, validated by an ETag from its `updated_at` (not Last-Modified, which the - sidecar fallback would answer with a stale 304 after a delete) so a matching - `If-None-Match` is a 304 without reading the image; sidecar and embedded art keep - `max-age=86400`. Only GET/HEAD of streaming-shaped - paths skip the request timeout, so the cover upload stays bounded. - Error codes `book_not_found`, `invalid_override` (+ `field`), `too_large`, - `unsupported_image`. The internal book id appears only inside the opaque cursor. - **Community covers** (`api/handlers_community_covers.go`): a match candidate's `cover_url` is - an image on its own host (Audible's CDN, Open Library), which the console's CSP can't load, so - the server fetches it (`meta.Service.FetchCover`: http(s) only, every connection's resolved - address checked public at dial time, redirects included (`publicAddr`; no proxy), <= 3 - redirects, 15 s, `maxConcurrentCoverFetches`, never holding `coverReads`). - `POST /admin/meta/covers` (`{urls, size}`, <= 12) returns thumbnails as `data:` URLs in - order (`""` = couldn't be fetched or decoded; cached in the thumbnail cache by URL, a failed - fetch not; each holds `communityReads` from its fetch to the end of its decode; asks for the - same thumbnail at once share one fetch, `communityFlights`); `PUT /admin/libraries/{id}/cover/community?path=` (`{url}`) keeps one as the book's - custom cover through `saveCustomCover`, the upload's own path, once `keepableCover` has read its - header (413 past `media.MaxThumbnailSourcePixels`), re-encoded within 1600 px first when it is - over 5 MiB (audited `book.cover_set`, `source: community`); 400 for a URL - that isn't http(s), 502 `cover_unavailable`, 413/415 as an upload. Both 404 `metadata_off` - while metadata is off (`metadataOff`). Tests swap the fetch through `API.fetchCover`. - **The player's community covers** (rail entries, "Previous books", the recording) come - through `GET /libraries/{id}/meta/cover?path=&url=&size=160|320|640` (`handleMetaCover`, - capability `meta_covers` = `metadataOn()`; media auth, so `?token=`): the web player's - CSP takes images only from the server, which stays so (`TestHTMLCSPImagesStaySameOrigin`), - and a listener's device never contacts a cover host. Not an open proxy: the path is - scope-checked like `/meta` (the same `bookEnvelope` lookup), and `url` must be one the book's cached envelope hands out - (`meta.Enrichment.HandsOutCover`: the recording's or any rail view's entry, exact match), - else 404 before any fetch. That beats a remembered-URL set (lost on restart, needs a - bound) and a signed URL (a new wire field on every cover, a key to keep): the envelope - is already cached in memory and in `meta_cache`, so the check is a lookup. Served only as - a JPEG thumbnail through `communityThumbnail` (same cache key as the console's batch, - `FetchCover`'s guards, `communityReads`), so upstream bytes and content types never - reach the client; ETag `"community--"`, `coverCache` (a day). - 404 `no such cover` (unmatched, untagged, URL not handed out), 502 metadata unavailable, - 502 `cover_unavailable` (fetch failed, not cached but remembered per URL for - `communityRetryAfter`, 1 min, so a broken or hung host isn't fetched on every render; the - console's batch ignores that and always retries), 404 `no cover` (not an image, over - the pixel bound). It stays under the request timeout (`isStreamingPath` matches - only `/libraries/{id}/cover|stream` exactly, `isLibraryMedia`). - `POST /admin/covers` (`api/handlers_covers.go`, Phase 2b) is how the console shows - covers: `{books:[{library_id,path}], size: 160|320|640}` (<= 60) returns JPEG - thumbnails as `data:` URLs in request order (`""` = no art), resolved like - `/cover` (custom, sidecar via `SafeJoin`, embedded; never on-demand indexing), from - one `catalog.CoverSources` query per library for the whole batch. - One request per page of covers instead of one per cover, no token in any URL, - ~20 KB a cover instead of full art. - `media.Thumbnail` refuses sources over `MaxThumbnailSourcePixels` from the header - (decompression bombs), `media.ThumbCache` is a byte-bounded LRU keyed by the art's - version (custom `updated_at`, file size + mtime) holding the raw JPEG (the admin - batch base64-encodes it), and - `coverReads` bounds the art being read or waiting to be decoded and `thumbSem` the - decodes, both across requests. The player gets - the same thumbnails from `GET /libraries/{id}/cover?size=160|320|640` (capability - `cover_sizes`; one code path, `coverArt` + `coverThumbnail`; ETag = size + art version, - 304 on a match; custom `no-cache`, file art `max-age=86400`; any other size 400). - `cover_version` (`catalog.CoverVersion`, a 10-char hash of `books.cover_art`) is set - from index data whenever a book is indexed (custom stamp, else mtime, size and - sidecar path; `SetCover`, `DeleteCover`, moves and joins recompute it). Every - thumbnail (a cache hit too, a 304 revalidation included) records on a book what - it lacks for the art it read: `cover_color` (`media.CoverPalette` on the scaled image: - dominant bucket = `bg`, the most vibrant bucket nudged in HSL lightness to WCAG 4.5:1 - against it = `accent`, else none; `on_accent` white/black) and, with it, the art's - own version as `cover_art`, so `cover_version` follows a sidecar replaced in place - and equals the thumbnail ETag's hash; `catalog.RecordCoverColors` is - compare-and-set on the identity it was read under and bounded (250 ms, detached - from the request). Both are on the player `Book` JSON (`omitempty`). Admin book rows (list and book page) also carry - `cover_color` (the Series spines) and `matched` (the `matched=` filter's rule), and `POST /admin/shares/{id}/paths` - also takes `{"rules":[...]}` (<= 1000, one transaction) for adding a selection. -- **Rate limiting by route class** (`rateLimit` in `api/middleware.go`, buckets in - `api/ratelimit.go`), read off the handler `mux.Handler` picks: static files - (`web.IsStatic`) are not counted; media routes (`mediaAuth`, behind `requireBookMediaAuth` for - cover/stream and `requireMediaAuth` for `/meta/cover`, whose `mediaHandler` marks them) limit - themselves: refused while the address's - general bucket is empty (`Ready`), a failed authentication charges it (`Charge`: - even past empty, down to -burst, so requests that passed `Ready` together all pay), an - authenticated one spends from `mediaLimiter` (~200/s, burst 2000, per credential id) - and charges the address's bucket when that refuses it (the lookup was already done); - everything else spends from `ipLimiter` (~50/s, burst 200, per IP). The brute-force - lockouts (`limiter`) are separate. Tests: `ratelimit_routes_test.go`. -- **Job queue, scan history, schedules, ignore rules (admin redesign Phase 3, `library/jobs.go`, - `schedule.go`, `ignore.go`, `problems.go`)**: every scan (startup, schedule, admin rescan, a - library or folder-setting change) goes through `Scanner.Enqueue`; `Scanner.Start` runs ONE - worker (scans never compete for the disk or the single DB writer) and the scheduler. Coalescing: - a library already waiting returns the waiting job; one running gets one follow-up for a manual or - change trigger (the running scan may have read the old setting) but not for a schedule or - startup. Each run is a `scan_runs` row (`catalog/scanruns.go`: trigger, who, status - `running|ok|partial|unavailable|failed|cancelled|interrupted`, counts (a move counts once, as - moved), a log of at most 300 `RunEvent`s (problems take at most half, so the moves and removed - paths after them keep room; past the cap events are only counted); 100 kept per library; rows - left `running` by a stopped server become `interrupted` at startup). `Cancel` drops a queued job - or cancels the running scan's context: discovery and the per-book loop stop on it, and a scan - stops before its prune, never inside it (the prune, `DeleteBooksNotIn`, is one transaction run - to completion once begun). Requested scans are capped at an hour (`jobContext`); the startup - scan, a library's full index, isn't. A root that doesn't answer the bounded root probe stops at - the unavailable guard instead of holding the one worker. Deleting a library cancels its scans - (`CancelLibrary`). **Prune semantics are unchanged**: a vanished - book is deleted from the index (no ghost rows) and its path lands in the run log; durable state - is path-keyed, so progress comes back if the files do. Schedules (`ParseSchedule`): `""`, - `every:{1,3,6,12,24}h` after the newest run started, `daily:HH:MM` in the server's zone; the - newest run of any trigger counts, so a server that was off runs a missed daily scan on return; - a due slot dropped without a run of its own (cancelled while queued, or folded into a scan of - the library already running) counts as a start too (`jobQueue.skipped`), so it isn't re-queued. - Ignore rules (`NormalizeIgnore`/`ParseIgnore`) live in the DB because the server never writes to - the library: gitignore-lite (no `/` = a name at any depth, a `/` before the end, a leading one - included, = anchored to the root, trailing `/` = folders only, case-insensitive, `#` comments, - <= 100 patterns of <= 200 bytes); the scan, `BrowseFS` and `IndexPath` all honour them, and rules - that skip every indexed book prune it rather than tripping the unavailable guard. - `metadata.Extract` reports `OpenErr`/`ProbeErr` - (ffprobe runs with `-v error` so its message survives, path prefix stripped); `noteProblem` - records the first read problem per book (`unreadable|empty_file|probe_failed`), and a scan - re-reads an otherwise unchanged book once the file its problem names opens or probes again - (`problemCleared`: a fixed permission changes neither mtime nor size). A folder book - whose parts are all >= 1 h with >= 2 distinct titles (tag, else file name) gets - `suspect_parts`; rows from before 0017 are checked once with a tag read. `PATCH - /admin/libraries/{id}` rescans only when the root or ignore rules change. -- **Health issues (`catalog/issues.go`, `api/handlers_health.go`)**: computed from the index on - request; each book kind is one SQL predicate (`issuePredicates`) shared by the counts and by - `GET /admin/books?issue=` (and `&issue_ignored=true`), so they can't disagree: `no_cover` - (checked rows only), `no_chapters` (<= 1 chapter, > 2 h), `unmatched` (left out of the summary - while metadata is off), `transcode`, `suspect` (not a folder with a `book` override, the - category's own fix), `scan_error`, `series_spellings` (`catalog/seriesspellings.go`: another book of - its library spells its main series another way, same `series_key`; listed per series by `GET - /admin/issues/series-spellings[?library_id=&ignored=true]` -> `{groups: [{library_id, suggested, - spellings: [{name, books}], books}]}`, from the same predicate, `suggested` being `seriesName`; - open = a book not ignored, ignored view = a book ignored; the console's fix is the author merge's bulk - `series` edit), `split_discs` (one book split across disc folders, listed by - its first disc: `books.split_parent` set, no override on that folder; the console's fix sets - `book` on it, which joins them; its discs never also group as duplicates). `split_parent` is - worked out during discovery (`library.markSplitDiscs`: a folder book whose folder is a disc - set, the same `discSets` predicate the join uses, so the fix always joins; `IndexPath` the - same from the folder's subtree) and stored by `UpsertBook`; an unchanged book whose value - differs (a sibling came or went, a pre-0022 row) gets it via `SetSplitParent` without a - reindex. `duplicate` is groups within one - library (`DuplicateGroups`: same fingerprint AND size, same ASIN/ISBN (whatever the lengths), - or `metaKey` with a length within 1 min / 2% (`sameLength`), an unknown length matching only - another unknown one; copies across libraries are deliberate). Each join marks its kind on one - copy and a group's `reason` is the strongest its joins support (`dupSet.reason`): - `id_length_differs` (a group a shared ASIN/ISBN joined whose shortest and longest known - lengths `sameLength` rejects, the whole group's so an unknown-length book can't bridge two - recordings; an unknown length is no evidence either way); outranks even identical audio, as the group holds a different - recording, and the console then calls no copy the one to keep) > `same_files` > `same_id` > - `same_book`. A group is hidden while every member is ignored, so a new copy brings it back. Endpoints (admin only): `GET /admin/issues` (counts + samples + offline - libraries with books/listeners + `checked_at`), `GET /admin/issues/duplicates` (the open - groups, or with `?ignored=true` only the ignored ones), `POST`/`DELETE /admin/issues/ignore` (<= 1000), `POST - /admin/libraries/{id}/book/rescan?path=` (`IndexPath` now, on a context detached from the - request timeout so a slow re-read is still saved; 404 `not_indexable`), `GET - /admin/jobs` (running + queued + schedules), `DELETE /admin/jobs/{id}`, `POST /admin/scan` - (queue every library: `Scanner.EnqueueAll`, also the startup scans), `GET /admin/scan-runs` - (`?library_id=&before=&limit=`, `next_before`) and `GET /admin/scan-runs/{id}` (with the log). - Codes `invalid_schedule`, `invalid_pattern`, `not_indexable`. -- **Sessions, devices, Activity stats (admin redesign Phase 4a, `catalog/sessions.go`, - `activity.go`, `progress_admin.go`, `auth/devices.go`, `api/handlers_activity.go`)**: players name - themselves in `X-AudioSilo-Client: [/] [()]` (`auth.ParseClient`, strict; - the console sends `AudioSilo Admin (web)`, the player `AudioSilo/ ()`, and only - same-origin on web). `authenticate` calls `auth.ResolveRequest` (or, for the `?token=` of a book's - cover or stream with no `Authorization` header, `auth.ResolveCastRequest`; both are `auth.resolve`), - which records the app (only when - the request names one, so a headerless `