fix: local-path contract for Lexicon/Engine, one-way NAS sync, and cleanup - #49
Open
rancur wants to merge 2 commits into
Open
fix: local-path contract for Lexicon/Engine, one-way NAS sync, and cleanup#49rancur wants to merge 2 commits into
rancur wants to merge 2 commits into
Conversation
…eanup The library had drifted across two roots and Engine DJ had lost its database. Three compounding failures, all now fixed at the source rather than papered over with a periodic repair job. 1. The worker wrote finished audio to the share ROOT, not Database/. 339 files across 189 artist folders ended up outside the library Lexicon and Engine read. MUSIC_LIBRARY_PATH is now /music/Database and is wired through docker-compose (it was hardcoded there, so setting it in .env did nothing). The bind mount deliberately still points at the share root so Plex path translation and the ~4,300 existing /music/Database/... file_path rows keep resolving -- remapping the mount instead would have broken every one of them. 2. Lexicon was handed /Volumes/* paths, which Engine DJ refuses. That, not SMB, is why new tracks never reached Engine: 40 rows carried a /Volumes/Macintosh HD/ symlink prefix while only 3 were actually on SMB. lexicon_library_path is now /Users/willcurran/Music/Database. 3. Synology Drive two-way synced the whole music share. It conflict-copied Engine's live 595 MB SQLite database 12 times (~9.4 GB) and jammed permanently on SoundSwitch project files. Replaced with scripts/sync-nas-to-mac.sh: one-way NAS->Mac, so conflict copies are structurally impossible. SSH-side change detection plus SMB transfer takes 1.3 s per pass versus 5m46s for a full SMB scan, with a 6 h reconcile as the safety net. Local paths cost replication lag, so tasks/sync_gate.py holds each import until the file has landed on the Mac, keyed off a heartbeat the sync agent publishes. It fails open on every degenerate case -- a gate that can deadlock the pipeline is worse than the lag it prevents. 10 tests cover that contract. scripts/repoint-lexicon-local.sh gains local-mode execution and the ~/Music/<Artist> -> ~/Music/Database/<Artist> mapping, and is marked a one-shot: it fixed this once on 2026-07-13 and the problem returned because nothing automated it. Now that new tracks arrive canonical, finding work again is a regression signal. Also in this change: - scripts/consolidate-share-root.py: the root -> Database/ migration, non- destructive on collisions (byte-identical copies are quarantined, differing ones kept side by side). - scripts/dedupe-report.py: read-only duplicate/quality analysis. Separates redundant Lexicon rows from real duplicate files from archive upgrade candidates, and detects remix descriptors in the parent folder as well as the filename so different mixes are not reported as duplicates. - Delete MatcherService/DownloaderService/VerifierService (463 lines, referenced nowhere; that logic lives in sync-worker/tasks/). - main.py read its version from /app/VERSION instead of a hardcoded "2.1.0". - admin.py compared versions as strings, so "2.9.0" > "2.10.1" was True and the update banner was inverted across exactly the boundary this deploy sat on. - CodeQL scanned Python only; sync-web's TypeScript was never analysed. - deep-repair.sh computed a verdict then dispatched to a commented-out stub, so it could never have alerted anyone. Wired to WAXFLOW_ALERT_WEBHOOK. - backup-lexicon.sh removed (self-documented no-op; backup-lexicon-db.sh is real). - bump-version.sh now updates every version source, not just VERSION. - CLAUDE.md documents the path contract and corrects "no test suite exists". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HmUiLHPmKoz215WAWV5eHe
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
|
|
||
| def _db(**config) -> str: | ||
| path = tempfile.mktemp(suffix=".db") |
Two problems found by running the finished system, not by reading it. 1. LEXICON RE-ADDS THE SYMLINK PREFIX. The path-contract fix stops WaxFlow emitting /Volumes paths, but it is NOT sufficient: Lexicon canonicalises an imported location through the boot-volume symlink, rewriting the clean /Users/willcurran/Music/Database/... path WaxFlow hands it into /Volumes/Macintosh HD/Users/... — which Engine DJ refuses. Proven live: the first import after the cutover (id 11288, 2026-08-09 16:26) was stored with that prefix despite WaxFlow sending a clean /Users path. This is the true origin of the 40 Engine-invisible rows, and it means repoint-lexicon-local.sh is a recurring chore, not the one-shot the previous commit claimed. Since it cannot be prevented at the source, make it impossible to MISS: a read-only (mode=ro, safe while Lexicon runs) count of /Volumes/* locations on every sync pass, logged loudly with the fix command. This exact drift went unnoticed from March to August and cost the Engine library. 2. THE HEARTBEAT WAS FILLING #recycle. The share has Synology's recycle bin enabled, and writing the heartbeat as write-tmp-then-mv made every replace a deletion — one #recycle entry every 120 s, forever. Truncate in place instead. A torn read is harmless: sync_gate.py fails open on unparseable JSON by design. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HmUiLHPmKoz215WAWV5eHe
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The library had drifted across two roots and Engine DJ had lost its database. Three compounding failures, fixed at the source rather than papered over with a periodic repair job.
1. The worker wrote to the share ROOT, not
Database/. 339 audio files across 189 artist folders ended up outside the library that Lexicon and Engine actually read.MUSIC_LIBRARY_PATHis now/music/Database, and it's wired throughdocker-compose.yml— it was hardcoded there, so setting it in.envdid nothing.The bind mount deliberately still points at the share root. Remapping it instead would have broken all ~4,300 existing
/music/Database/...file_pathrows and Plex's path translation. This was the one non-obvious call in the change.2. Lexicon was handed
/Volumes/*paths, which Engine DJ refuses. That — not SMB — is why new tracks never reached Engine: 40 rows carried a/Volumes/Macintosh HD/symlink prefix while only 3 were actually on SMB.lexicon_library_pathis now/Users/willcurran/Music/Database.3. Synology Drive two-way synced the entire music share. It conflict-copied Engine's live 595 MB SQLite database 12 times (~9.4 GB) and jammed permanently retrying SoundSwitch project files the server rejects. Replaced with
scripts/sync-nas-to-mac.sh— one-way NAS→Mac, so conflict copies are structurally impossible.How
sync-nas-to-mac.shis a hybrid: change detection over SSH (the NAS walks its own disk, ~0.9 s) and transfer over SMB. A full SMB scan takes 5m46s; an incremental pass takes 1.3 s. A 6 h reconcile is the safety net. Rsync-over-SSH isn't available — DSM refusesrsync --serverfor non-admin users.Local paths cost replication lag, so
tasks/sync_gate.pyholds each import until the file has landed on the Mac, keyed off a heartbeat the sync agent publishes intoInput/. It fails open on every degenerate case (missing, malformed, stale, or error heartbeat) and bounds the hold — a gate that can deadlock the pipeline is worse than the lag it prevents. 10 tests pin that contract down.scripts/repoint-lexicon-local.shgains local-mode execution and the~/Music/<Artist>→~/Music/Database/<Artist>mapping. It's now marked a one-shot: it fixed this once on 2026-07-13 and the problem returned because nothing automated it. Now that new tracks arrive canonical, finding work again is a regression signal, not routine maintenance.New tooling
scripts/consolidate-share-root.py— the root →Database/migration. Non-destructive on collisions: byte-identical copies are quarantined rather than deleted, differing ones kept side by side with a__fromrootsuffix.scripts/dedupe-report.py— read-only duplicate/quality analysis. Separates redundant Lexicon rows from real duplicate files from archive upgrade candidates, and reads remix descriptors from the parent folder as well as the filename, so a Maduk remix isn't reported as a duplicate of the original.Cleanup
MatcherService/DownloaderService/VerifierService— 463 lines referenced nowhere; that logic lives insync-worker/tasks/.main.pyhardcodedversion="2.1.0"whileVERSIONsaid2.10.1; now reads/app/VERSION.admin.pycompared versions as strings, so"2.9.0" > "2.10.1"wasTrue— the update banner was inverted across exactly the boundary this deployment was sitting on.deep-repair.shcomputed a repair verdict then dispatched to a commented-out stub, so it could never have alerted anyone. Wired toWAXFLOW_ALERT_WEBHOOK.backup-lexicon.shdeleted (self-documented no-op;backup-lexicon-db.shis the real one).bump-version.shnow updates every version source, not justVERSION./Users/openclaw/spotify-lexicon-sync/path in the backup plist.CLAUDE.mdgains a path-contract section and no longer claims "no test suite exists" (there are 22 test modules).Not included
A CodeQL change adding
javascript-typescript(sync-web's ~2,700 lines of TS are unscanned) is held back — theghtoken lacksworkflowscope. Patch is at~/WaxFlow-Backups/codeql-typescript.patch.Verification
sync_gatetests: 10/10 pass./Users/...paths, zero/Volumes/*.docker compose restartstuck for 5h43m, predating this work) and needs a daemon restart before the new images can run.🤖 Generated with Claude Code
https://claude.ai/code/session_01HmUiLHPmKoz215WAWV5eHe