v2.11.0 — path contract, one-way replication, and several bugs affecting every install

.env.example

@@ -17,5 +17,23 @@ NEXT_PUBLIC_API_URL=http://localhost:8402
 # CORS origins for API
 CORS_ORIGINS=http://localhost:8400
 
-# Music library path (inside container)
-MUSIC_LIBRARY_PATH=/music
+
+# --- Lexicon host paths -------------------------------------------------------
+# Where the machine running Lexicon sees the music tree. The worker rewrites its
+# own container path to this before telling Lexicon where a file is.
+#
+# Default (seeded in the DB) is the SMB mount, which needs no replication:
+#   LEXICON_LIBRARY_PATH=/Volumes/music/Database
+#   LEXICON_INPUT_PATH=/Volumes/music/Input
+#
+# If you ALSO export to Engine DJ, prefer LOCAL paths on the Lexicon host and run
+# scripts/sync-nas-to-mac.sh to replicate. See "Local paths and Engine DJ" in the
+# README. Contains a username, so it must be set per install:
+#   LEXICON_LIBRARY_PATH=/Users/you/Music/Database
+#   LEXICON_INPUT_PATH=/Users/you/Music/Input
+
+# --- Library root inside the container ----------------------------------------
+# The worker writes finished audio here and index_library scans it. Point it at a
+# SUBDIRECTORY of the bind mount, not the mount root, so the share root stays
+# clean and Plex path translation keeps working.
+MUSIC_LIBRARY_PATH=/music/Database

CHANGELOG.md

@@ -1,5 +1,88 @@
 # Changelog
 
+## 2.11.0 — Path contract, one-way replication, and a pile of real bugs
+
+Everything here was found by *running* the system during a live incident, not by
+reading it. Several are bugs any deployment would hit.
+
+### Fixed — bugs that affect every install
+- **`MUSIC_LIBRARY_PATH` was hardcoded in `docker-compose.yml`.** Both services
+  pinned `/music`, so setting it in `.env` silently did nothing. Now
+  `${MUSIC_LIBRARY_PATH:-/music/Database}`. Pointing the library root at a
+  SUBDIRECTORY of the bind mount keeps the share root clean and leaves Plex path
+  translation intact — remapping the mount instead would break every existing
+  `file_path` row.
+- **The update banner was inverted.** `admin.py` compared versions as strings, so
+  `"2.9.0" > "2.10.1"` was `True` and no update was ever offered across a
+  major.minor boundary. Replaced with a numeric-tuple compare.
+- **The API reported the wrong version.** `main.py` hardcoded `2.1.0` while
+  `VERSION` said `2.10.1`. It now reads `/app/VERSION`, baked in at build time.
+- **`busy_timeout` contradicted the connect timeout.** 5 s against `timeout=30`,
+  and `busy_timeout` is what governs — so `PATCH /api/settings` still returned
+  `{"detail":"database is locked"}` whenever the worker was mid-index. Both 30 s.
+- **`deep-repair.sh` could never alert anyone.** It computed a repair verdict and
+  then dispatched to a commented-out example. Wired to `WAXFLOW_ALERT_WEBHOOK`.
+- **CodeQL scanned Python only**; `sync-web`'s TypeScript went unanalysed. (Held
+  back from this PR — needs a `workflow`-scoped token.)
+
+### Added — one-way NAS -> Mac replication (optional)
+`scripts/sync-nas-to-mac.sh` + LaunchAgent. Pull-only, so conflict copies are
+structurally impossible. Hybrid transport: change detection over SSH (~0.9 s,
+the NAS walks its own disk) with an SMB transfer, versus 5 m 46 s for a full SMB
+scan; 6 h reconcile as the safety net. `tasks/sync_gate.py` holds each import
+until the file has landed locally, and **fails open** on every degenerate case —
+a gate that can deadlock the pipeline is worse than the lag it prevents.
+
+### Added — tools
+- `scripts/merge-duplicate-lexicon-rows.py` — Engine DJ's `Track.path` is UNIQUE,
+  so Lexicon rows beyond the distinct-file count can NEVER sync and can leave a
+  part-applied sync failing with `FOREIGN KEY constraint failed`. Migrates
+  playlist memberships onto the surviving row *before* deleting; a plain delete
+  would have destroyed 849 memberships on the library this was written against.
+- `scripts/consolidate-share-root.py` — moves stray artist folders from a share
+  root into the library root. Non-destructive on collisions.
+- `scripts/dedupe-report.py` — read-only duplicate/quality analysis. Reads remix
+  descriptors from the parent folder as well as the filename, so different mixes
+  are not reported as duplicates.
+
+### Fixed — macOS agent robustness
+- `ensure-music-mount.sh` treated **any** failed `ls` as a stale handle and
+  unmounted. From a launchd agent the share cannot be remounted (`mount volume`
+  has no keychain access; `mount_smbfs` returns `Authentication error`), so it
+  destroyed a working mount that only a human in Finder could restore. It now
+  never unmounts by default, and distinguishes macOS TCC's `Operation not
+  permitted` — which means the mount is fine and *this process* is denied — from
+  a real stale handle.
+- The share is addressed by IP/hostname, **not** a Bonjour service-instance name.
+  `NAME._smb._tcp.local` resolves only via service discovery; when that goes
+  stale, mounts hang forever instead of failing.
+- `osascript mount volume` is now watchdog-wrapped. It blocks indefinitely
+  waiting on a credential dialog that never appears under launchd, which wedged
+  both agents and stopped replication entirely.
+
+### Changed
+- Host-specific values moved out of the scripts into `~/.waxflow/waxflow.conf`.
+- `lexicon_library_path` / `lexicon_input_path` seed to the SMB default and can be
+  set via `LEXICON_LIBRARY_PATH` / `LEXICON_INPUT_PATH` — no username in defaults.
+- `bump-version.sh` updates every version source, not just `VERSION`.
+- README gains "Local paths and Engine DJ", including the two things that cost the
+  most time here: never put an Engine library inside a two-way sync, and Engine
+  holds at most one row per file.
+
+### Removed
+- `sync-api/services/{matcher,downloader,verifier}.py` — 463 lines referenced
+  nowhere; that logic lives in `sync-worker/tasks/`.
+- `scripts/backup-lexicon.sh` — self-documented no-op; `backup-lexicon-db.sh` is
+  the real one.
+
+### Corrected
+An earlier diagnosis held that Engine DJ refuses `/Volumes/*` locations. **It does
+not.** Tested against a real Engine library: all 40 rows carrying a
+`/Volumes/Macintosh HD/` prefix were present. That prefix is a symlink to `/` and
+resolves fine. The missing-tracks symptom was caused by two-way sync destroying
+the Engine database, not by path format.
+
+
 ## 2.10.0 — Sleep-tolerance catch-up: rescue downloaded-but-not-imported tracks
 
 Closes the last sleep/wake gap that stranded freshly-downloaded tracks with a real

CLAUDE.md

@@ -35,10 +35,15 @@ docker compose up -d
 ```
 
 ## Testing
-No test suite exists yet. This is a priority backlog item. When tests are added:
+22 test modules exist across `sync-worker/tests/` and `sync-api/tests/`.
 ```bash
-python3 -m pytest tests/ -v
+cd sync-worker && python3 -m pytest tests/ -v     # or: python3 -m unittest discover tests
+cd sync-api    && python3 -m pytest tests/ -v
 ```
+Most worker tests import `tasks.helpers`, which pulls in `spotipy` — run them inside
+the worker image (`docker exec waxflow-worker python3 -m pytest tests/`) or in a venv
+with `sync-worker/requirements.txt` installed. Leaf modules with no heavy imports
+(e.g. `tasks/sync_gate.py`) run anywhere.
 
 ## Coding Standards
 - Python 3.12, type hints encouraged
@@ -66,3 +71,43 @@ Key variables (see `.env.example` or README for full list):
 - **Pipeline stages**: new -> matching -> downloading -> verifying -> organizing -> complete
 - **5-layer dedup**: ISRC file index, Lexicon DB lookup, on-disk scan, Tidal ISRC, Tidal metadata
 - **Scan mode vs Full mode**: scan mode only matches existing library; full mode downloads new tracks
+
+## The path contract (READ THIS BEFORE TOUCHING PATHS)
+Rewritten 2026-08-08 after the library ended up split across two roots and Engine DJ
+lost its entire database. Three rules, and they are load-bearing:
+
+1. **The library root is `Database/`, not the share root.**
+   `MUSIC_LIBRARY_PATH=/music/Database` — the worker writes there and `index_library`
+   scans there. The bind mount deliberately still points at the share ROOT
+   (`MUSIC_HOST_PATH=/volume1/music`) so Plex path translation
+   (`plex_music_container_prefix=/music` -> `/volume1/music`) and the ~4,300 existing
+   `/music/Database/...` `tracks.file_path` rows keep resolving. Do NOT "simplify"
+   this by remapping the mount — it would break every one of those rows.
+
+2. **Prefer LOCAL Lexicon-host paths over `/Volumes/*`.**
+   `lexicon_library_path=<local path>` (e.g. `~/Music/Database`), set via
+   `LEXICON_LIBRARY_PATH`. NOT because Engine DJ rejects `/Volumes/*` — it does
+   not; that was tested on 2026-08-09 and disproved, all 40 rows carrying a
+   `/Volumes/Macintosh HD/` prefix were present in Engine. Prefer local paths
+   because an SMB path breaks the moment the share unmounts and the file then
+   exists only on the NAS. Engine stores paths relative to its own folder
+   (`../Database/<Artist>/...`), so `Engine Library/` must remain a sibling of
+   `Database/`.
+
+   Engine also enforces `UNIQUE (path)`: if Lexicon holds more rows than distinct
+   files, the surplus can never sync. See `scripts/merge-duplicate-lexicon-rows.py`.
+
+3. **Replication is one-way NAS -> Mac, and imports wait for it.**
+   `scripts/sync-nas-to-mac.sh` (launchd, 120 s) pulls `Database/` and `Input/` down;
+   `tasks/sync_gate.py` holds each import until the file has landed. Synology Drive is
+   NOT involved — two-way syncing the whole share produced 12 conflict copies of
+   Engine's database and jammed permanently on SoundSwitch project files.
+
+`scripts/repoint-lexicon-local.sh` normalises legacy rows. Note that Lexicon
+CANONICALISES imported paths through the boot-volume symlink, so new imports may
+reappear as `/Volumes/Macintosh HD/Users/...` — cosmetic, not a fault. The sync
+agent logs an action-needed line only for genuine SMB paths.
+
+**Never put an Engine library inside a two-way sync.** Two-way syncing the share
+containing Engine's live `m.db` produced 12 conflict copies and left the real
+database with 2 tracks in it. That, not path format, is what lost the library.

README.md

@@ -393,6 +393,69 @@ volumes:
 
 ---
 
+---
+
+## Local paths and Engine DJ
+
+By default WaxFlow tells Lexicon where a file is using the **SMB mount**
+(`/Volumes/music/...`). That needs no replication and works out of the box.
+
+If you also export from Lexicon to **Engine DJ**, there are two things worth
+knowing, both learned the hard way:
+
+**1. Never put your Engine library inside a two-way sync.** Engine's `m.db` is a
+live SQLite file. Two-way syncing the share that contains it produced 12 conflict
+copies (~9.4 GB) and left the real database with 2 tracks in it. Engine stores
+track paths *relative* to its own folder (`../Database/<Artist>/...`), so a copy
+on the NAS is meaningless as well as dangerous. Keep `Engine Library/` local and
+excluded from every sync.
+
+**2. Engine holds at most one row per file.** Its schema declares
+`CONSTRAINT C_path UNIQUE (path)`. If Lexicon has more rows than distinct files —
+several rows pointing at the same audio — the surplus can never sync, and a
+partially-applied sync can fail with `SqliteError: FOREIGN KEY constraint failed`.
+Check with:
+
+```sql
+SELECT COUNT(*), COUNT(DISTINCT location) FROM Track;   -- Lexicon's main.db
+```
+
+If those differ, `scripts/merge-duplicate-lexicon-rows.py` merges them safely — it
+migrates playlist memberships onto the surviving row *before* deleting, because a
+plain delete silently drops memberships the duplicate held.
+
+### Optional: local replication
+
+To hand Lexicon local paths instead, set `LEXICON_LIBRARY_PATH` /
+`LEXICON_INPUT_PATH` to paths on the Lexicon host and run the replication agent:
+
+```bash
+mkdir -p ~/.waxflow
+cp scripts/sync-nas-to-mac.sh scripts/ensure-music-mount.sh ~/.waxflow/
+chmod +x ~/.waxflow/*.sh
+printf 'WAXFLOW_SHARE_HOST=192.168.1.50
+WAXFLOW_NAS_SSH=nas
+' > ~/.waxflow/waxflow.conf
+sed "s|/Users/willcurran|$HOME|g" scripts/com.waxflow.sync-database.plist \
+    > ~/Library/LaunchAgents/com.waxflow.sync-database.plist
+launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.waxflow.sync-database.plist
+```
+
+It replicates **one way, NAS to Mac** — a pull-only replica cannot create conflict
+copies. `tasks/sync_gate.py` then holds each import until the file has actually
+landed locally, so Lexicon never imports a path that does not exist yet.
+
+Two macOS gotchas the scripts now handle explicitly, both of which cost real
+downtime here:
+
+* **Full Disk Access.** A launchd agent cannot read `/Volumes/*` until you add
+  `/bin/bash` under System Settings → Privacy & Security → Full Disk Access.
+  Without it every pass fails with `Operation not permitted`, which looks exactly
+  like a dead mount but is not.
+* **Address the NAS by IP or plain hostname, never a Bonjour *service* name.**
+  `NAME._smb._tcp.local` resolves only via service discovery; when that
+  advertisement goes stale, mounts **hang forever** rather than failing.
+
 ## Tech Stack
 
 | Component | Technology |

VERSION

@@ -1 +1 @@
-2.10.1
+2.11.0

docker-compose.yml

@@ -21,7 +21,11 @@ services:
       - SPOTIFY_REDIRECT_URI=${SPOTIFY_REDIRECT_URI:-http://localhost:8402/api/spotify/callback}
       - TIDARR_URL=${TIDARR_URL:-http://localhost:8484}  # optional: only for legacy Tidarr fallback
       - LEXICON_API_URL=${LEXICON_API_URL:-http://localhost:48624}
-      - MUSIC_LIBRARY_PATH=/music
+      # The bind mount above still points at the share ROOT so Plex path
+      # translation (plex_music_container_prefix=/music) and existing
+      # /music/Database/... file_path rows keep resolving. The LIBRARY root
+      # is a subdirectory of it -- that is what the worker writes into.
+      - MUSIC_LIBRARY_PATH=${MUSIC_LIBRARY_PATH:-/music/Database}
       - SLS_DB_PATH=/app/data/sync.db
     restart: unless-stopped
     healthcheck:
@@ -54,7 +58,11 @@ services:
       - SPOTIFY_REDIRECT_URI=${SPOTIFY_REDIRECT_URI:-http://localhost:8402/api/spotify/callback}
       - TIDARR_URL=${TIDARR_URL:-http://localhost:8484}  # optional: only for legacy Tidarr fallback
       - LEXICON_API_URL=${LEXICON_API_URL:-http://localhost:48624}
-      - MUSIC_LIBRARY_PATH=/music
+      # The bind mount above still points at the share ROOT so Plex path
+      # translation (plex_music_container_prefix=/music) and existing
+      # /music/Database/... file_path rows keep resolving. The LIBRARY root
+      # is a subdirectory of it -- that is what the worker writes into.
+      - MUSIC_LIBRARY_PATH=${MUSIC_LIBRARY_PATH:-/music/Database}
       - SLS_DB_PATH=/app/data/sync.db
       - TIDDL_PATH=/tiddl-config
     depends_on:

scripts/backup-lexicon-db.sh

@@ -44,11 +44,12 @@
 
 set -euo pipefail
 
-LEXICON_SSH="${LEXICON_SSH:-willcurran@192.168.1.116}"
+# Set LEXICON_SSH to <user>@<lexicon-host>, or "local" when running ON that Mac.
+LEXICON_SSH="${LEXICON_SSH:-local}"
 LEXICON_DB="${LEXICON_DB:-\$HOME/Library/Application Support/lexicon/main.db}"
 MAC_BACKUP_DIR="${MAC_BACKUP_DIR:-\$HOME/WaxFlow-Backups/lexicon-db}"
 NAS_SSH="${NAS_SSH:-nas}"
-NAS_BACKUP_DIR="${NAS_BACKUP_DIR:-/volume1/homes/willcurran/WaxFlow-Backups/lexicon-db}"
+NAS_BACKUP_DIR="${NAS_BACKUP_DIR:-/volume1/homes/$USER/WaxFlow-Backups/lexicon-db}"
 KEEP="${KEEP:-14}"
 LOG_DIR="${LOG_DIR:-$HOME/.waxflow/logs}"
 SKIP_ON_HYPERBACKUP="${SKIP_ON_HYPERBACKUP:-1}"

scripts/bump-version.sh

@@ -1,12 +1,45 @@
 #!/bin/bash
 # Usage: ./scripts/bump-version.sh 1.2.1
-VERSION=$1
+#
+# Single source of truth is the VERSION file. Everything that needs a version
+# either reads it at runtime (sync-api/main.py reads /app/VERSION, baked in by the
+# Dockerfile) or is updated here. Previously only VERSION was bumped, so main.py
+# reported "2.1.0" while VERSION said 2.10.1, and the MusicBrainz user agent was
+# stuck at 2.9.
+set -euo pipefail
+
+VERSION="${1:-}"
 if [ -z "$VERSION" ]; then
-    echo "Usage: $0 <version>"
+    echo "Usage: $0 <version>" >&2
     exit 1
 fi
+if ! echo "$VERSION" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+$'; then
+    echo "REFUSE: '$VERSION' is not semver (X.Y.Z)" >&2
+    exit 1
+fi
+
+REPO_DIR="$(cd "$(dirname "$0")/.." && pwd)"
+cd "$REPO_DIR"
+
+if ! grep -q "^## $VERSION" CHANGELOG.md 2>/dev/null; then
+    echo "WARNING: CHANGELOG.md has no '## $VERSION' section yet." >&2
+    printf "Continue anyway? [y/N] "
+    read -r reply
+    case "$reply" in [yY]*) ;; *) echo "aborted"; exit 1;; esac
+fi
+
 echo "$VERSION" > VERSION
-git add VERSION
+
+# MusicBrainz asks that the user agent identify the app version (it is how they
+# rate-limit and contact you); keep the seeded default in step with the release.
+MAJOR_MINOR="$(echo "$VERSION" | cut -d. -f1,2)"
+sed -i.bak -E "s|('musicbrainz_user_agent', 'WaxFlow/)[0-9]+\.[0-9]+|\1$MAJOR_MINOR|" sync-api/init_db.py
+rm -f sync-api/init_db.py.bak
+
+echo "--- files changed ---"
+git --no-pager diff --stat VERSION sync-api/init_db.py
+
+git add VERSION sync-api/init_db.py
 git commit -m "Bump version to $VERSION"
 git tag "v$VERSION"
 git push && git push --tags

scripts/com.openclaw.waxflow-lexicon-backup.plist

@@ -22,7 +22,7 @@
 	<key>ProgramArguments</key>
 	<array>
 		<string>/bin/bash</string>
-		<string>/Users/openclaw/spotify-lexicon-sync/scripts/backup-lexicon-db.sh</string>
+		<string>/Users/openclaw/waxflow/scripts/backup-lexicon-db.sh</string>
 	</array>
 	<key>RunAtLoad</key>
 	<false/>