diff --git a/CHANGELOG.md b/CHANGELOG.md index 383c4ab..9087539 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,48 @@ All notable changes to Library Manager will be documented in this file. +## [0.9.0-beta.153] - 2026-06-12 + +### Security +- **#255: Validate and sanitize BookDB response data** — Added `_sanitize_api_response()` to strip null bytes, control characters, HTML tags, and truncate oversized fields from all Skaldleita/BookDB API responses. Prevents path traversal, XSS, and memory issues from malformed responses. +- **#256: Signing nonce and tighter replay window** — Added UUID nonce (`X-LM-Nonce`) to signed request headers for replay attack prevention. Reduced timestamp tolerance from 300s to 120s. Backwards compatible with existing BookDB. +- **#236: SSRF prevention on plugin endpoints** — (from 0.9.0-beta.152) + +### Fixed +- **#252: Rate limiter aligned with actual BookDB tiers** — Increased min_delay from 3.6s (1000/hr) to 12.0s (300/hr) to match BookDB's API key tier. Added adaptive backoff from Retry-After headers and specific 429 handling in audio identification. +- **#253: server_notice handled in audio identification** — BookDB's abort/notice signals are now processed in `identify_audio_with_bookdb()` (submit, poll, and result stages), matching existing coverage in `search_bookdb()`. + +### Enhanced +- **#254: Differentiated BookDB source trust weighting** — `sl_source` field now maps to distinct source names (`bookdb_audio`, `bookdb_transcript`, `bookdb_scrape`) with graduated weights (65/55/45) instead of treating all BookDB audio results identically. + +### Documentation +- **#257: ABS plugin architecture** — Added `docs/ABS-Plugin-Architecture.md` scoping Library Manager as an Audiobookshelf metadata provider plugin. + +--- + +## [0.9.0-beta.152] - 2026-06-12 + +### Security +- **#219: Library boundary enforcement** on undo/replace/remove file operations +- **#237: XSS fix** — escapeHtml covers all 5 HTML entities including single quotes +- **#236: SSRF prevention** on custom plugin endpoints + +### Fixed +- **#220: Atomic status guard** on apply_fix prevents race condition double-renames +- **#221: Worker stop checks** in all 4 legacy batch loops +- **#222-224: Rate limits and circuit breakers** for BookDB contribute, community consensus, Gemini language detect, OpenRouter identify, plus process endpoint re-entrancy guard +- **#225-226: Undo parent mkdir** and path sanitize false positive fix +- **#227: L1 preserves original metadata** on validation failure +- **#228-229: Pipeline stuck prevention** — L3 trust_sl no longer deletes queue entry before L4, AI verify creates history on null drastic check +- **#230: Custom source weights** now actually used in confidence calculations +- **#231: Fixed broken route** in queue multi-edit modal +- **#232: Settings save** now persists sl_trust_mode and enable_isbn_lookup, removed duplicate field +- **#233: Double-click prevention** on all 11 history page action buttons +- **#234: Famous numeric titles** (1984, 2001, etc.) no longer flagged as garbage +- **#235: Confidence scale normalization** handles both 0-1 and 0-100 scales + +--- + ## [0.9.0-beta.151] - 2026-06-05 ### Changed diff --git a/README.md b/README.md index 71417fa..18f3d8e 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ **Smart Audiobook Library Organizer with Multi-Source Metadata & AI Verification** -[![Version](https://img.shields.io/badge/version-0.9.0--beta.152-blue.svg)](CHANGELOG.md) +[![Version](https://img.shields.io/badge/version-0.9.0--beta.153-blue.svg)](CHANGELOG.md) [![Docker](https://img.shields.io/badge/docker-ghcr.io-blue.svg)](https://ghcr.io/deucebucket/library-manager) [![License](https://img.shields.io/badge/license-AGPL--3.0-blue.svg)](LICENSE) diff --git a/app.py b/app.py index dfee0f1..6b53fb0 100644 --- a/app.py +++ b/app.py @@ -11,7 +11,7 @@ - Multi-provider AI (Gemini, OpenRouter, Ollama) """ -APP_VERSION = "0.9.0-beta.152" +APP_VERSION = "0.9.0-beta.153" GITHUB_REPO = "deucebucket/library-manager" # Your GitHub repo # Versioning Guide: diff --git a/docs/ABS-Plugin-Architecture.md b/docs/ABS-Plugin-Architecture.md new file mode 100644 index 0000000..199c24b --- /dev/null +++ b/docs/ABS-Plugin-Architecture.md @@ -0,0 +1,252 @@ +# Audiobookshelf Plugin Architecture for Library Manager + +## Overview + +Library Manager as a metadata provider plugin for Audiobookshelf (ABS). When ABS needs to identify or enrich audiobook metadata, it queries Library Manager, which leverages the full Skaldleita pipeline: GPU Whisper transcription, 50M+ book database matching, multi-source API lookups, and AI-powered consensus verification. + +**Current state:** LM has an `abs_client.py` that pulls data FROM ABS (listening progress, library items, user management). This document covers the reverse direction — making LM available TO ABS as a metadata agent. + +--- + +## Data Flow + +``` +Audiobookshelf Library Manager Skaldleita (BookDB) + │ │ │ + │ 1. "Identify this book" │ │ + │ ─────────────────────────> │ │ + │ (title, author, audio clip) │ │ + │ │ 2. Audio → Whisper queue │ + │ │ ─────────────────────────> │ + │ │ │ + │ │ 3. Metadata match (50M DB) │ + │ │ <───────────────────────── │ + │ │ │ + │ │ 4. API enrichment │ + │ │ (Audnexus, Google Books, │ + │ │ OpenLibrary, Hardcover) │ + │ │ │ + │ │ 5. AI consensus verify │ + │ │ (Gemini / OpenRouter) │ + │ │ │ + │ 6. Enriched metadata │ │ + │ <───────────────────────── │ │ + │ (author, title, narrator, │ │ + │ series, year, confidence) │ │ +``` + +--- + +## Architecture Options + +### Option A: LM as ABS Metadata Provider (Recommended) + +ABS has a metadata provider plugin system. LM registers as a provider that ABS queries during its "Match" and "Quick Match" flows. + +**How it works:** +- ABS sends search queries to LM's API +- LM runs its full pipeline (BookDB + multi-source + AI verification) +- LM returns structured metadata in ABS's expected format +- ABS users see LM results alongside other providers (Google Books, Audible, etc.) + +**LM endpoints needed:** +``` +GET /api/abs/search?query=&author=<author> + → Returns ABS-formatted search results + +GET /api/abs/match?title=<title>&author=<author>&narrator=<narrator> + → Returns single best match with confidence + +POST /api/abs/identify-audio + → Accepts audio clip, returns identification via Skaldleita Whisper + +GET /api/abs/cover?title=<title>&author=<author> + → Returns cover image URL if available +``` + +**Pros:** Native ABS integration, users see LM in their provider list +**Cons:** ABS metadata provider API has a specific contract we must match exactly + +### Option B: LM as Standalone Enrichment Service + +LM runs alongside ABS and periodically scans the ABS library for books with missing/low-quality metadata, then pushes corrections back via ABS API. + +**How it works:** +- LM uses existing `abs_client.py` to read ABS library +- Identifies books with missing narrators, series info, etc. +- Runs its pipeline to find correct metadata +- Pushes updates back to ABS via API + +**Pros:** Works without ABS plugin system, LM controls the schedule +**Cons:** Delayed updates, LM needs write access to ABS + +### Option C: Hybrid (Both A + B) + +Register as metadata provider AND run background enrichment. Provider handles new additions; background scan catches existing gaps. + +--- + +## Recommended Approach: Option A (Metadata Provider) + +### ABS Metadata Provider Contract + +ABS expects providers to implement these operations: + +```json +// Search request +GET /search?query=Brandon+Sanderson+Mistborn + +// Search response +{ + "matches": [ + { + "title": "The Final Empire", + "subtitle": "Mistborn Book 1", + "author": "Brandon Sanderson", + "narrator": "Michael Kramer", + "publisher": "Macmillan Audio", + "publishedYear": "2006", + "description": "...", + "cover": "https://...", + "isbn": "9780765350381", + "asin": "B002V1O7UE", + "series": [ + { "series": "Mistborn", "sequence": "1" } + ], + "language": "en", + "duration": 95940 + } + ] +} +``` + +### LM → ABS Field Mapping + +| ABS Field | LM Source | Notes | +|-----------|-----------|-------| +| `title` | BookProfile.title | Highest confidence value | +| `author` | BookProfile.author | Consensus from all sources | +| `narrator` | BookProfile.narrator | Often from audio credits (L2) or Audnexus | +| `series[].series` | BookProfile.series | From BookDB or API lookups | +| `series[].sequence` | BookProfile.series_num | Position in series | +| `publishedYear` | BookProfile.year | From API lookups | +| `cover` | External API | Audnexus or Google Books cover URL | +| `isbn` | API lookups | Google Books or OpenLibrary | +| `asin` | Audnexus | Audible ASIN if available | +| `duration` | Audio file analysis | From ffprobe or ABS library item | +| `language` | Gemini detection | Audio language detection (L2) | +| `description` | API enrichment | Google Books or OpenLibrary | + +### Confidence Translation + +LM uses 0-100 confidence scores. ABS doesn't have a native confidence field, but we can: +1. Only return results above a configurable threshold (default: 60) +2. Sort results by confidence (best first) +3. Include confidence in a custom field for display + +### New LM Files + +``` +library_manager/abs_provider.py # ABS metadata provider Blueprint + - /api/abs/search # Search endpoint + - /api/abs/match # Best-match endpoint + - /api/abs/identify # Audio identification endpoint + - /api/abs/covers/<isbn> # Cover proxy +``` + +### Authentication + +Options: +1. **Shared secret** — LM generates an API key, user enters it in ABS custom provider config +2. **Token exchange** — ABS sends its API token, LM validates via ABS `/api/me` endpoint +3. **None (local only)** — If both run on same machine, trust localhost + +Recommend: Shared secret (simple, works across networks). + +### Configuration (LM Settings Page) + +Add to the Integrations tab: + +``` +ABS Metadata Provider +├── Enable as ABS metadata source: [toggle] +├── Provider API Key: [auto-generated, copyable] +├── Minimum confidence threshold: [slider 0-100, default 60] +├── Include audio identification: [toggle, default on] +└── Provider URL for ABS config: http://<lm-host>:5757/api/abs +``` + +--- + +## Security Considerations + +### What This Opens +- A new API surface (`/api/abs/*`) accessible from the network +- Audio clip uploads from ABS to LM +- LM's metadata forwarded to ABS (already LM's core function) + +### Mitigations +- API key authentication on all `/api/abs/*` endpoints +- Rate limiting (reuse existing rate limiter infrastructure) +- Audio clip size limits (same as existing Skaldleita submission limits) +- Input validation on all query parameters (sanitize before pipeline) +- SSRF protection already in place for plugin endpoints (issue #236) +- No new filesystem access — ABS provider only reads/returns metadata + +### What NOT to Expose +- Direct Skaldleita API passthrough (LM should be the intermediary, not a proxy) +- BookDB API keys or signing credentials +- Internal BookProfile objects (serialize to ABS format only) +- Direct database access + +--- + +## Implementation Phases + +### Phase 1: Search Provider +- New Blueprint: `abs_provider.py` +- `/api/abs/search` endpoint using existing `search_bookdb()` + API lookups +- API key auth +- Settings UI toggle +- Basic test with ABS custom provider + +### Phase 2: Audio Identification +- `/api/abs/identify` endpoint +- Accept audio clips from ABS +- Route through Skaldleita Whisper pipeline +- Return structured results + +### Phase 3: Background Enrichment (Option B hybrid) +- Scan ABS library via existing `abs_client.py` +- Queue items with missing metadata +- Push enrichments back to ABS +- Configurable schedule (daily/weekly) + +### Phase 4: Cover Art + Descriptions +- `/api/abs/covers/<isbn>` proxy endpoint +- Cache covers locally +- Pull descriptions from Google Books / OpenLibrary enrichment + +--- + +## ABS Custom Provider Setup (User-Facing) + +In ABS Settings → Providers → Custom: +``` +Name: Library Manager +Base URL: http://<lm-ip>:5757/api/abs +API Key: <paste from LM settings> +``` + +Then in ABS, when matching a book: +1. Click "Match" on a library item +2. Select "Library Manager" as provider +3. See results from LM's multi-source pipeline +4. One-click apply metadata + +--- + +## Related Issues +- #257 — This feature +- #252-#256 — Integration fixes that improve the pipeline feeding this provider +- abs_client.py — Existing read-only ABS integration (complements this) diff --git a/library_manager/models/book_profile.py b/library_manager/models/book_profile.py index 89450ae..3d0c93b 100644 --- a/library_manager/models/book_profile.py +++ b/library_manager/models/book_profile.py @@ -18,7 +18,10 @@ 'id3': 80, # Embedded by producer/publisher 'json': 75, # Explicit metadata file 'nfo': 70, # Release info files - 'bookdb': 65, # Verified database match + 'bookdb': 65, # Verified database match + 'bookdb_audio': 65, # BookDB database match (same as bookdb) + 'bookdb_transcript': 55, # BookDB Whisper transcript only (unverified) + 'bookdb_scrape': 45, # BookDB live web scrape (unreliable) 'ai': 60, # AI verification 'audnexus': 55, # Audible data 'googlebooks': 50, # Google Books API diff --git a/library_manager/pipeline/layer_audio_id.py b/library_manager/pipeline/layer_audio_id.py index 5cdbde1..7babd58 100644 --- a/library_manager/pipeline/layer_audio_id.py +++ b/library_manager/pipeline/layer_audio_id.py @@ -714,18 +714,20 @@ def process_layer_1_audio( c = conn.cursor() # Update book with audio-identified info - sl_source = result.get('sl_source', 'audio_transcription') + # Use the mapped source name (bookdb_audio/bookdb_transcript/bookdb_scrape) + # for proper trust weighting in BookProfile, falling back to sl_source + profile_source = result.get('source', result.get('sl_source', 'audio_transcription')) base_confidence = 85 if confidence == 'high' else 70 profile = { - 'author': {'value': author, 'source': sl_source, 'confidence': base_confidence}, - 'title': {'value': title, 'source': sl_source, 'confidence': base_confidence}, + 'author': {'value': author, 'source': profile_source, 'confidence': base_confidence}, + 'title': {'value': title, 'source': profile_source, 'confidence': base_confidence}, } if narrator: - profile['narrator'] = {'value': narrator, 'source': sl_source, 'confidence': 80} + profile['narrator'] = {'value': narrator, 'source': profile_source, 'confidence': 80} if series: - profile['series'] = {'value': series, 'source': sl_source, 'confidence': 75} + profile['series'] = {'value': series, 'source': profile_source, 'confidence': 75} if series_num: - profile['series_num'] = {'value': str(series_num), 'source': sl_source, 'confidence': 75} + profile['series_num'] = {'value': str(series_num), 'source': profile_source, 'confidence': 75} # Issue #227: Save original author/title BEFORE updating the DB. # If validation fails later, we must revert these to prevent diff --git a/library_manager/providers/bookdb.py b/library_manager/providers/bookdb.py index 372c928..0cb79f9 100644 --- a/library_manager/providers/bookdb.py +++ b/library_manager/providers/bookdb.py @@ -11,6 +11,7 @@ """ import os +import re import time import logging import subprocess @@ -26,6 +27,8 @@ record_api_success, handle_rate_limit_response, API_CIRCUIT_BREAKER, + API_RATE_LIMITS, + API_RATE_LOCK, ) from library_manager.utils.voice_embedding import ( is_voice_embedding_available, @@ -50,6 +53,33 @@ def get_and_clear_server_abort(): _abort_state.notice = None return notice +def _sanitize_api_response(data, context='bookdb'): + """Sanitize string fields from API responses to prevent path traversal, XSS, and oversized data.""" + if not isinstance(data, dict): + return data + + MAX_FIELD_LENGTH = 500 + sanitized = {} + string_fields = ('title', 'author', 'author_name', 'narrator', 'series', 'series_name', 'name', 'variant', 'edition') + + for key, value in data.items(): + if key in string_fields and isinstance(value, str): + original = value + # Strip null bytes and control characters (keep newlines for descriptions) + value = re.sub(r'[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]', '', value) + # Strip HTML tags + value = re.sub(r'<[^>]+>', '', value) + # Truncate oversized fields + if len(value) > MAX_FIELD_LENGTH: + value = value[:MAX_FIELD_LENGTH] + # Log if sanitization changed the value + if value != original: + logger.warning(f"[{context}] Sanitized '{key}': {original[:100]!r} → {value[:100]!r}") + sanitized[key] = value + + return sanitized + + # Skaldleita API endpoint (our metadata service, legacy name: BookDB) BOOKDB_API_URL = "https://bookdb.deucebucket.com" # URL unchanged for backwards compatibility # Public API key for Library Manager users (no config needed) @@ -76,27 +106,30 @@ def get_user_agent(): # Request signing - uses shared module for Skaldleita sync # See library_manager/signing.py for constants and derivation logic # Skaldleita fetches that file to stay in sync automatically -from library_manager.signing import generate_signature +from library_manager.signing import generate_signature, generate_nonce def get_signed_headers(): """ Generate signed headers for Skaldleita API requests. - Returns dict with User-Agent, X-LM-Signature, and X-LM-Timestamp. + Returns dict with User-Agent, X-LM-Signature, X-LM-Timestamp, and X-LM-Nonce. Secret is derived from version - changes with each release. + Nonce prevents replay attacks within the timestamp tolerance window. Skaldleita fetches signing.py to stay in sync. See library_manager/signing.py for derivation logic. """ timestamp = str(int(time.time())) lm_version = get_lm_version() - signature = generate_signature(lm_version, timestamp) + nonce = generate_nonce() + signature = generate_signature(lm_version, timestamp, nonce=nonce) return { 'User-Agent': f'LibraryManager/{lm_version}', 'X-LM-Signature': signature, 'X-LM-Timestamp': timestamp, + 'X-LM-Nonce': nonce, } @@ -147,7 +180,7 @@ def search_bookdb(title, author=None, api_key=None, retry_count=0, bookdb_url=No logger.debug(f"Skaldleita: Circuit open, skipping ({remaining}s remaining)") return None - rate_limit_wait('bookdb') # 3.6s delay = max 1000/hr, never skips + rate_limit_wait('bookdb') # 12.0s delay = max 300/hr (API key tier), never skips # Use configured URL or fall back to default cloud URL base_url = bookdb_url or BOOKDB_API_URL @@ -169,6 +202,19 @@ def search_bookdb(title, author=None, api_key=None, retry_count=0, bookdb_url=No # Handle rate limiting with exponential backoff if resp.status_code == 429: rl = handle_rate_limit_response(resp, 'bookdb', retry_count) + # Adaptive: if BookDB says slow down via Retry-After, increase our + # min_delay so future requests are paced more conservatively + if rl['retry_after']: + try: + retry_secs = int(rl['retry_after']) + with API_RATE_LOCK: + current_delay = API_RATE_LIMITS['bookdb']['min_delay'] + if retry_secs > current_delay: + API_RATE_LIMITS['bookdb']['min_delay'] = float(retry_secs) + logger.info(f"[BOOKDB] Adaptive rate limit: increased min_delay " + f"from {current_delay}s to {retry_secs}s per Retry-After header") + except (ValueError, TypeError): + pass if rl['should_retry']: time.sleep(rl['wait_seconds']) return search_bookdb(title, author, api_key, retry_count + 1, bookdb_url, @@ -184,6 +230,7 @@ def search_bookdb(title, author=None, api_key=None, retry_count=0, bookdb_url=No API_CIRCUIT_BREAKER['bookdb']['failures'] = 0 data = resp.json() + data = _sanitize_api_response(data) # Issue #208: honor Skaldleita server_notice. Log every notice; on # action=abort_task, stash in thread-local so the watch-folder worker @@ -226,6 +273,9 @@ def search_bookdb(title, author=None, api_key=None, retry_count=0, bookdb_url=No if not best_book: best_book = books[0] + if best_book: + best_book = _sanitize_api_response(best_book) + # Build result - handle standalone books (no series) and series books result = { 'title': best_book.get('title') if best_book else (series.get('name') if series else None), @@ -368,12 +418,45 @@ def identify_audio_with_bookdb(audio_file, extract_seconds=90, bookdb_url=None): timeout=30 # Just submitting, should be fast ) + # Handle 429 rate limiting specifically — record the failure and + # adaptively increase min_delay if Retry-After header is present + if response.status_code == 429: + rl = handle_rate_limit_response(response, 'bookdb') + if rl['retry_after']: + try: + retry_secs = int(rl['retry_after']) + with API_RATE_LOCK: + current_delay = API_RATE_LIMITS['bookdb']['min_delay'] + if retry_secs > current_delay: + API_RATE_LIMITS['bookdb']['min_delay'] = float(retry_secs) + logger.info(f"[SKALDLEITA] Adaptive rate limit: increased min_delay " + f"from {current_delay}s to {retry_secs}s per Retry-After header") + except (ValueError, TypeError): + pass + logger.warning(f"[SKALDLEITA] Rate limited (429) on audio identify") + return None + if response.status_code != 200: logger.warning(f"[SKALDLEITA] API returned {response.status_code}: {response.text[:200]}") return None submit_data = response.json() + # Issue #253: Check server_notice on the submit response itself. + # BookDB may signal abort before we even enter the poll loop. + notice = submit_data.get('server_notice') + if notice: + code = notice.get('code', 'unknown') + msg = notice.get('message', '') + upgrade_url = notice.get('upgrade_url') + severity = notice.get('severity', 'info') + logger.warning(f"[SKALDLEITA] server notice ({severity}) [{code}]: {msg}") + if upgrade_url: + logger.warning(f"[SKALDLEITA] upgrade: {upgrade_url}") + if notice.get('action') == 'abort_task': + _abort_state.notice = notice + return None + # Check if it's the new queue system (has ticket_id) or old sync system if 'ticket_id' in submit_data: # New queue system - poll for result @@ -415,6 +498,21 @@ def identify_audio_with_bookdb(audio_file, extract_seconds=90, bookdb_url=None): # Got result! data = status_data.get('result', {}) logger.info(f"[SKALDLEITA] Complete! Processing result...") + + # Issue #253: Check server_notice in the poll result + notice = status_data.get('server_notice') + if notice: + code = notice.get('code', 'unknown') + msg = notice.get('message', '') + upgrade_url = notice.get('upgrade_url') + severity = notice.get('severity', 'info') + logger.warning(f"[SKALDLEITA] server notice ({severity}) [{code}]: {msg}") + if upgrade_url: + logger.warning(f"[SKALDLEITA] upgrade: {upgrade_url}") + if notice.get('action') == 'abort_task': + _abort_state.notice = notice + return None + break elif status == 'error': @@ -433,6 +531,21 @@ def identify_audio_with_bookdb(audio_file, extract_seconds=90, bookdb_url=None): # Old sync system - response has result directly data = submit_data + # Issue #253: Check server_notice in the final result data. + # Covers the old sync path and acts as a catch-all for the queue path. + notice = data.get('server_notice') + if notice: + code = notice.get('code', 'unknown') + msg = notice.get('message', '') + upgrade_url = notice.get('upgrade_url') + severity = notice.get('severity', 'info') + logger.warning(f"[SKALDLEITA] server notice ({severity}) [{code}]: {msg}") + if upgrade_url: + logger.warning(f"[SKALDLEITA] upgrade: {upgrade_url}") + if notice.get('action') == 'abort_task': + _abort_state.notice = notice + return None + # Process result (same for both systems) transcript = data.get('transcript') or '' matched_books = data.get('matched_books') or [] @@ -448,18 +561,28 @@ def identify_audio_with_bookdb(audio_file, extract_seconds=90, bookdb_url=None): sl_source = data.get('source', 'audio') # 'database', 'audio', or 'live_scrape' requeue_suggested = data.get('requeue_suggested', False) + # Map sl_source to differentiated source names for trust weighting + # database = verified DB match (highest), audio = Whisper only, live_scrape = web scraping + source_map = { + 'database': 'bookdb_audio', + 'audio': 'bookdb_transcript', + 'live_scrape': 'bookdb_scrape', + } + mapped_source = source_map.get(sl_source, 'bookdb_audio') + result = { 'author': data.get('author') or (best_match.get('author_name') if best_match else None), 'title': data.get('title') or (best_match.get('title') if best_match else None), 'narrator': data.get('narrator'), 'series': best_match.get('series_name') if best_match else None, 'series_num': best_match.get('series_position') if best_match else None, - 'source': 'bookdb_audio', - 'sl_source': sl_source, # Where SL got the data: 'database' or 'audio' + 'source': mapped_source, + 'sl_source': sl_source, # Where SL got the data: 'database', 'audio', or 'live_scrape' 'requeue_suggested': requeue_suggested, # True if LM should retry later 'confidence': 'high' if best_match or sl_source == 'database' else 'medium', 'transcript': transcript[:500], } + result = _sanitize_api_response(result, context='SKALDLEITA') if result['author'] and result['title']: logger.info(f"[SKALDLEITA] Identified: {result['author']} - {result['title']}" + @@ -468,7 +591,7 @@ def identify_audio_with_bookdb(audio_file, extract_seconds=90, bookdb_url=None): if transcript: logger.info(f"[SKALDLEITA] No match but got transcript ({len(transcript)} chars) - returning for AI fallback") - return {'transcript': transcript, 'source': 'bookdb_audio'} + return {'transcript': transcript, 'source': mapped_source} logger.warning(f"[SKALDLEITA] No identification and no transcript returned") return None @@ -626,6 +749,7 @@ def lookup_community_consensus(title, author=None, bookdb_url=None): __all__ = [ 'BOOKDB_API_URL', 'BOOKDB_PUBLIC_KEY', + '_sanitize_api_response', 'get_signed_headers', 'search_bookdb', 'identify_audio_with_bookdb', diff --git a/library_manager/providers/fingerprint.py b/library_manager/providers/fingerprint.py index 518e875..e04d4ec 100644 --- a/library_manager/providers/fingerprint.py +++ b/library_manager/providers/fingerprint.py @@ -18,6 +18,7 @@ import requests +from library_manager.providers.bookdb import _sanitize_api_response from library_manager.providers.rate_limiter import handle_rate_limit_response logger = logging.getLogger(__name__) @@ -157,6 +158,7 @@ def lookup_fingerprint( # Return book if linked to database, otherwise contributed metadata result = data.get('book') or data.get('contributed_metadata') if result: + result = _sanitize_api_response(result, context='FINGERPRINT') result['confidence'] = data.get('confidence', 0) title = result.get('title', 'Unknown') logger.info(f"[FINGERPRINT] Match found: {title} " diff --git a/library_manager/providers/rate_limiter.py b/library_manager/providers/rate_limiter.py index efcac67..7c1465e 100644 --- a/library_manager/providers/rate_limiter.py +++ b/library_manager/providers/rate_limiter.py @@ -11,9 +11,9 @@ # Rate limiting to stay under API limits # Format: api_name -> {last_call, min_delay} -# Skaldleita (bookdb): 3.6 sec delay = 1000 requests/hour max, spread evenly (never hits limit) +# Skaldleita (bookdb): 12.0 sec delay = 300 requests/hour max (API key tier) API_RATE_LIMITS = { - 'bookdb': {'last_call': 0, 'min_delay': 3.6}, # Skaldleita: 3600s / 1000 = 3.6s between calls = exactly 1000/hr + 'bookdb': {'last_call': 0, 'min_delay': 12.0}, # Skaldleita: 3600s / 300 = 12.0s between calls = 300/hr (API key tier) 'audnexus': {'last_call': 0, 'min_delay': 2.0}, # ~100/hr community API - be nice 'openlibrary': {'last_call': 0, 'min_delay': 1.5}, # They request max 1/sec, add buffer 'googlebooks': {'last_call': 0, 'min_delay': 1.0}, # 1000/day, no per-sec limit but be safe @@ -39,7 +39,7 @@ def rate_limit_wait(api_name): """ Wait if needed to respect rate limits for the given API. - For Skaldleita: 3.6s delay ensures exactly 1000 requests/hour max. + For Skaldleita: 12.0s delay ensures max 300 requests/hour (API key tier). All requests go through - no skipping, just proper pacing. """ with API_RATE_LOCK: diff --git a/library_manager/signing.py b/library_manager/signing.py index b8d081c..e484b7a 100644 --- a/library_manager/signing.py +++ b/library_manager/signing.py @@ -10,6 +10,7 @@ import hashlib import hmac import time +import uuid # Signing salt - combined with version to derive per-release secret # Change this to invalidate ALL existing signatures (nuclear option) @@ -19,7 +20,7 @@ ACCEPTED_VERSION_COUNT = 5 # Timestamp tolerance in seconds (reject requests with old timestamps) -TIMESTAMP_TOLERANCE = 300 # 5 minutes +TIMESTAMP_TOLERANCE = 120 # 2 minutes def derive_secret(version: str) -> str: @@ -27,16 +28,23 @@ def derive_secret(version: str) -> str: return hashlib.sha256(f"{SIGNING_SALT}:{version}".encode()).hexdigest()[:32] -def generate_signature(version: str, timestamp: str) -> str: - """Generate HMAC signature for a request.""" +def generate_nonce() -> str: + """Generate a random nonce (UUID4 hex) to prevent request replay.""" + return uuid.uuid4().hex + + +def generate_signature(version: str, timestamp: str, nonce: str = None) -> str: + """Generate HMAC signature for a request. Nonce is optional for backwards compatibility.""" secret = derive_secret(version) message = f"{timestamp}:{version}" + if nonce: + message = f"{timestamp}:{version}:{nonce}" return hmac.new(secret.encode(), message.encode(), hashlib.sha256).hexdigest()[:32] -def verify_signature(signature: str, version: str, timestamp: str) -> bool: - """Verify a signature (for Skaldleita server-side use).""" - expected = generate_signature(version, timestamp) +def verify_signature(signature: str, version: str, timestamp: str, nonce: str = None) -> bool: + """Verify a signature (for Skaldleita server-side use). Nonce is optional for backwards compatibility.""" + expected = generate_signature(version, timestamp, nonce=nonce) return hmac.compare_digest(signature, expected) @@ -46,6 +54,7 @@ def verify_signature(signature: str, version: str, timestamp: str) -> bool: 'ACCEPTED_VERSION_COUNT', 'TIMESTAMP_TOLERANCE', 'derive_secret', + 'generate_nonce', 'generate_signature', 'verify_signature', ]