Repository navigation
Prepare open-source release: audit fixes, security hardening, CI and docs - #1
Merged
Merged
Conversation
Implement Google Drive, S3-compatible, Dropbox, OneDrive and WebDAV storage providers with AES-GCM encrypted credentials, background upload/sync/restore/cleanup tasks, and safety checks that refuse destructive operations without a verified local copy. Add the /admin/storage WebUI and 'streamvault storage' CLI, integrate auto-upload after recording and remux, extend the worker pool, and add Docker/supervisord worker packaging plus Nginx X-Accel aliases for archives, legacy files, VOD cache, thumbnails and HLS. Update install/docs, harden secret handling, redact provider errors, and add a 93-test regression suite.
WEBUI.webui starts a network-bound "twitch-badge-sync" daemon thread on import. The thread writes into whichever BASE_DIR is active and races TemporaryDirectory cleanup, causing intermittent "OSError: [Errno 39] Directory not empty" teardown errors (6/25 runs with test_storage.py alone, 0/25 with the sync disabled). Only the test environment changes (TWITCH_BADGE_AUTO_SYNC=0); no assertion is touched. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tzan3xULuUxadM3Azn5fzJ
Recorder: - stop_channel_and_release_lock(): release the channel flock only after the child process has exited, so a new recorder cannot start writing the same channel while the old FFmpeg is still running. Removes the duplicated reap/restart block in the main loop. - wait_for_process_exit(): replace unbounded process.wait() on FFmpeg with a hard deadline (FFMPEG_FINALIZE_TIMEOUT, default 30 s), then kill and reap. - disk_blocks_recording(): resolve not-yet-existing target paths to the nearest existing parent; log probe errors instead of swallowing them. - archive_is_shared(): skip duplicate cleanup when several sources feed one archive (fail closed). - Live-only monitor: honour shutdown_requested, use interruptible_sleep and define CHECK_INTERVAL (previously undefined -> NameError). Core: - claim_next_task(max_attempts_by_type=...): stale running tasks that exhausted their retry budget are marked 'failed' instead of being requeued forever. - Sentry: send_default_pii off and traces sampled at 0.0 by default (SENTRY_SEND_DEFAULT_PII, SENTRY_TRACES_SAMPLE_RATE), documented in .env.example. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tzan3xULuUxadM3Azn5fzJ
- delete_folder / empty_folder re-check at execution time and defer (TaskDeferred, 120 s) while a recording (.part, recording lock with a live PID, fresh current_recording pointer) or a remux lock is active; the WebUI check only ran at enqueue time. - empty_folder never removes *.part files, even if the check raced. - Remove "fuser -k -9" on archive folders: it could SIGKILL the recorder/FFmpeg writing into them. - _submit_claimed_task(): release the in-flight slot and defer the task when pool dispatch fails, so capacity does not leak. - Pass TASK_MAX_ATTEMPTS to claim_next_task(). - Transcription: log and fail fast when run_transcribe.py is missing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tzan3xULuUxadM3Azn5fzJ
- core/health.py: dependency-free parse_supervisor_status(); /healthz now reports missing supervisor programs and no longer expects the removed "livebot" program (optional if present). - WEBUI/webui.py: STREAMVAULT_IMPORT_ONLY=1 skips import-time side effects (admin bootstrap, badge sync thread); a failing admin bootstrap is now logged instead of silently ignored. - run_transcribe.py: standalone Whisper runner used by the worker. It was referenced by worker.process_transcribe() but missing from the tree, so ASR tasks could not run. Heavy ML imports stay out of the long-lived worker process. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tzan3xULuUxadM3Azn5fzJ
Covers lock ordering (stop before unlock, unlock on stop failure), bounded FFmpeg waits, disk-space guard on missing paths, shared-archive duplicate protection, worker folder-safety guards, dispatch capacity release, stale-task retry budget, supervisor status parsing, the run_transcribe helper / import-only guard, live-only monitor shutdown and network probe timeouts. fcntl is shimmed on Windows; the suite runs unchanged on Linux. Full suite: 126 tests (93 storage + 33 reliability), Linux 0 skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tzan3xULuUxadM3Azn5fzJ
install.sh and INSTALL.md configure Nginx on plain HTTP port 80. Secure-only session cookies are never sent back over HTTP, which makes the login redirect loop. Write VOD_SECURE_COOKIES=false into the generated env file and document switching it to true after enabling HTTPS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tzan3xULuUxadM3Azn5fzJ
scripts/streamvault_vps.sh (validate | deploy | rollback) checks what the Windows audit host cannot exercise: - runtime: venv Python >= 3.12, compile gate, import smoke test with STREAMVAULT_IMPORT_ONLY, optional unit tests as the service user (asserts the Linux-only symlink test runs instead of skipping) - filesystem: flock semantics, network-FS detection for SQLite WAL, archive symlinks (worker cleanup is fail-closed), www-data traversal along resolved paths, service UMask, disk-pause thresholds - supervisor: supervisorctl status parsed via core.health (same code as /healthz), socket access for the service user, recorder stopwaitsecs vs. FFmpeg finalize; systemd fallback with PrivateDevices and restart-loop checks - Nginx X-Accel: internal flag and alias target of every /_protected* location, disable_symlinks, cookie/TLS consistency, direct-access 404, plus an end-to-end round trip on a throwaway Nginx instance (production config is not modified) - HLS RAM disk: LIVE_HLS_DIR set, tmpfs, ownership, read/write permissions, free space, tmpfiles.d persistence (--fix) deploy snapshots the app, refuses to cut active recordings without --force, gates on compile + tests before restarting services and rolls back automatically on failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tzan3xULuUxadM3Azn5fzJ
Architecture map, essential commands and binding invariants (no silent failures, immutable tests, stop-before-unlock, bounded waits, SQLite WAL rules, worker file safety, ASR process isolation, storage hygiene). Records Python 3.12+ as the runtime floor, actual test counts (93 + 33) and the real disk-pause defaults (90 % / 20 GB). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tzan3xULuUxadM3Azn5fzJ
mobile_app_head_tags() nested an f-string containing "\n" inside an f-string expression. That is only valid from Python 3.12 (PEP 701), so WEBUI/webui.py did not compile on the distro Python of Ubuntu 22.04 (3.10) and Debian 12 (3.11) that install.sh targets - the WebUI would not start. The Tizen TV tags are now built in a variable first; the rendered HTML is unchanged (verified with is_tizen_tv True/False). - scripts/streamvault_vps.sh: Python floor back to 3.10 - CLAUDE.md: Python 3.10+ and a rule against PEP 701 syntax, which the 3.12 Windows audit host accepts silently Verified: all 45 files compile and 126/126 tests pass on Python 3.10.12. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tzan3xULuUxadM3Azn5fzJ
Index content is already LF; this keeps it that way independent of core.autocrlf. Shell scripts, bin/streamvault, *.conf and .env.example are always checked out with LF so files copied from a Windows checkout to the VPS/Docker image do not break on CRLF. Binary assets are marked binary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tzan3xULuUxadM3Azn5fzJ
supervisord defaulted to stopwaitsecs=10 for [program:recorder], which SIGKILLs FFmpeg before FFMPEG_FINALIZE_TIMEOUT (30 s) and truncates the running recording on every container restart. Set stopwaitsecs=60 and raise the compose stop_grace_period from 30 s to 75 s so Docker does not kill the container before supervisord's own grace period ends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The recorder defaulted to /dev/shm/streamrecorder_hls while the WebUI (core.config) serves from /dev/shm/streamvault_hls. Without an explicit LIVE_HLS_DIR the recorder wrote segments the WebUI never found. The recorder, chat_clip and live_watch_diag now all derive the path from config.LIVE_HLS_ROOT; live_watch_diag no longer hard-codes the old directory name when matching FFmpeg command lines. The VPS validator downgrades a missing LIVE_HLS_DIR from fail to warn, since both services now share the same default. Adds LiveHlsRootTests (3 tests). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
collect_health_status() only asked supervisorctl, so on systemd VPS installs checks.services.ok was always null and a dead recorder or worker never degraded /healthz. It also treated supervisorctl exit code 3 (some programs not RUNNING) as unknown instead of degraded. collect_service_status() now parses supervisorctl output on exit 0/3 and otherwise falls back to `systemctl show -p Id,LoadState,ActiveState,SubState streamvault-*.service`. Units are matched by Id; not-found units count as missing. Crash loops (activating/auto-restart) are degraded, transitional states such as the recorder's finalize window (deactivating) are not. ok stays null only when neither manager has units, and that case is logged. Adds SystemdHealthTests and ServiceStatusFallbackTests (8 tests). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Smart-TV mode was deployment-specific: it read a token from a hardcoded /root path, redirected TV user agents (or any ?tv=1 request) to /login/tv with that token in the URL, and logged them in as a fixed gatekeeper account that defaulted to the admin role. - drop is_tizen_tv(), the /login/tv route and the login auto-redirect - drop the is_tv template flag and all TV-only layout branches - drop tizen_tv.css / tizen_tv_nav.js and the TV checks in the live player and chat scripts - maintenance mode no longer exempts a default bypass user - add regression tests for the removed route and TV user agents Chromecast/AirPlay casting is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Casting needed cookie-free media URLs, so every VOD, clip, HLS and live URL carried a signed ?cast= bearer token (12 h, OBS variant 10 years) and media responses sent Access-Control-Allow-Origin: *. Anyone holding a copied link could stream without an account. - media, HLS, live and chat endpoints are session-only (media_login_required answers 401 instead of a login redirect) - drop token issue/verify helpers, wildcard media CORS and /api/live/cast-url - OBS browser-source pages now require a login; their chat and playlist requests send the session cookie - replace cast_playback.js with hls_support.js (native-HLS choice on Apple devices only) and disable the browsers' remote-playback UI - drop the cast buttons, cast CSS and the gstatic.com CSP entries - add regression tests for forged tokens, CORS and the OBS login Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
get_client_ip() took the left-most X-Forwarded-For hop, which the client controls: nginx's $proxy_add_x_forwarded_for only appends. Rotating that header bypassed Flask-Limiter, the login attempt counter and IP bans, and let an attacker get a victim's IP banned. Behind a trusted proxy the app now uses X-Real-IP (set by both shipped nginx configs) or the right-most X-Forwarded-For hop, validated as an IP. VOD_TRUSTED_PROXIES accepts CIDR networks; docker-compose trusts the private ranges of its network, since the app port is not published and all users otherwise shared the nginx container's IP for bans. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
core.tube_terms requires a frontmatter block with privacyVersion in content/datenschutz.md. The shipped file had none, so /datenschutz and /agb (which embeds the privacy version) both answered HTTP 500 on every fresh install. Drop the leading H1 as well; the renderer only handles H2/H3 and takes the title from the frontmatter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_is_current_recording() returned False on any error, and skipped half-written .current_recording*.json pointers. Its callers then treated a recording that was still being written as finished and remuxed it. os.kill() raising PermissionError for a recorder owned by another user was also read as "process gone". Unreadable pointers, PermissionError and directory errors now count as active and are logged. Only a removed pointer or a dead PID clears it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
load_json() returned the default on any exception. Favorites, history,
profiles, shares and user settings are read-modify-written, so a
truncated file silently came back as {} or [] and the next save replaced
the user's data for good.
A missing file still yields the default. A file that does not parse is
moved to <path>.corrupt-<ts> and logged before the default is returned;
other read errors are raised instead of being mistaken for "empty".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Folder deletion only checked that the target is a direct child of BASE_DIR. BASE_DIR defaults to the application directory when ./data is missing (and was /app in Docker), so deleting a streamer named "core", "WEBUI" or ".meta" removed application code or the database. config.is_reserved_archive_name() now rejects dot-names and the app's top-level directories; delete_folder and empty_folder tasks and the delete route refuse them and log the attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docker-compose ran with VOD_BASE_DIR=/app, so recordings, the SQLite database and caches were mixed into the bind-mounted source checkout. Data now lives in ./data (/app/data); the nginx X-Accel aliases follow. The protected nginx locations, the installer's live-HLS block and the INSTALL.md samples still sent Access-Control-Allow-Origin: * for casting, which no longer exists. Media is same-origin only. Existing Docker installs: move archives, streamrecorder.db and the dot-directories from the project root into ./data before upgrading. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Streamlink: the opt-in playlist proxy defaulted to a hardcoded list of third-party ad-free proxy hosts and /etc/streamrecorder/plugins. No proxy is built in anymore; TWITCH_PROXY_PLAYLIST must be set by the operator, otherwise nothing is added and a warning is logged. Remove the shadowed duplicate streamlink_twitch_auth_args(). - Backups: the list/download/delete routes defaulted to /root/streamrecorder_backups and "create" ran a script from a fixed /root path that is not part of this repo. BACKUP_DIR now defaults to <META_DIR>/backups and the script comes from VOD_BACKUP_SCRIPT. Backup deletion errors are logged instead of swallowed. - Remove the unused DISCORD_LOGS_DIR default (/root/Discord Bots/logs). - tests: pin tube_terms paths in LegalPagesTests, which failed when an isolated test imported the module while APP_DIR was patched. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The login page still pointed new users to a Discord invite (discord.gg/streamvault) that this project does not own, called itself "Cosmic Login" with an "N" logo and an "Enter Control" button, and said "Nur fuer unsere Community". Use neutral self-hosted wording, the "SV" mark used elsewhere, and plain titles for the dashboard and app shell. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Login: legacy `body.page-login h1` rules in style.css force clamp(42px, 7vw, 78px) with !important, so "STREAMVAULT" overflowed the 440px card. Scope the card heading with higher specificity (28px) and use normal case. - ASSET_VERSION was a hand-maintained constant (20261024-instant-launch) that nobody bumped, so browsers kept serving stale CSS/JS after updates. Derive it at startup from the files under WEBUI/static (path, size, mtime); VOD_ASSET_VERSION can still pin it. - tests: LegalPagesTests depended on a database another test happened to create; give it its own via IsolatedBaseTest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A container-overflow scan at 375px found content clipped by the app shell's overflow:hidden on four pages: - header action buttons did not wrap (admin pages) - user-management tabs and table were cut off instead of scrolling - fixed min-widths wider than a phone column: minmax(280px, 1fr) grids and inline min-width: 260px fields (storage, channels, search, files) Header actions wrap, RBAC tabs and table scroll horizontally, and the fixed minimums are capped with min(..., 100%). Scan after the fix: no overflow on 11 pages at 375, 768 and 1280px. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pyproject.toml pins ruff to py310 with F, E9, B and PLE: real defects only, no style churn. `ruff check .` now passes: - remove unused imports (onedrive, webdav, chat_clip, worker, tests) - remove unused locals and `as e` bindings in logged handlers - rename loop variables that shadowed flask.g - `raise ... from None` for re-raised ID validation errors - log the dashboard shm cache write failure instead of `except: pass` Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GitHub Actions runs compileall, ruff and the unittest suite on the production interpreters (3.10 Ubuntu 22.04, 3.11 Debian 12) plus 3.12, and `bash -n` on the installer, VPS script and CLI. Dependabot keeps pip and action versions current. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- CONTRIBUTING.md: dev setup, the CI checks, and the rules the code relies on (3.10 compatibility, no silent failures, fail closed around recordings, stop-before-unlock, path validation) - SECURITY.md: private vulnerability reporting via GitHub - CODE_OF_CONDUCT.md: Contributor Covenant 2.1 by reference - CHANGELOG.md: unreleased changes, breaking changes first with the manual steps operators need - issue forms (bug, feature), security contact link, PR checklist Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- describe only verified features: Twitch and YouTube (not Kick),
chat replay, live restream, clips, Whisper transcripts, cloud
archive, roles/groups/allowlists, OBS overlays
- drop unverifiable claims ("enterprise-grade", "100% sanitized",
"kernel-level") and emoji headings
- correct the Python badge to 3.10+, add CI badge, note German-only UI
- document Docker's ./data directory, HTTPS before exposure, and
VOD_TRUSTED_PROXIES
- add a legal notice on recording third-party streams
- CLAUDE.md: current test counts (167) and the ruff check
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FUNDING.yml enables GitHub's Sponsor button for buymeacoffee.com/blackwellcode; the README gets a badge in the header and a short "Support the project" section. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prepares StreamVault for its open-source release: reliability and security fixes from a full audit, removal of deployment-specific features, and the usual project infrastructure.
Breaking changes (details in CHANGELOG.md)
./data(VOD_BASE_DIR=/app/data). Existing installs must move archives,streamrecorder.dband the dot-directories before upgrading.?cast=media URLs, wildcard media CORS). Media is session-only./live/obs,/chat/obs) require a login./login/tvand the gatekeeper auto-login.TWITCH_PROXY_PLAYLIST; no third-party proxies are built in.BACKUP_DIR(default<VOD_META_DIR>/backups) andVOD_BACKUP_SCRIPT.Security
X-Real-IPor the right-mostX-Forwarded-Forhop; rotating the header no longer bypasses login throttling.VOD_TRUSTED_PROXIESaccepts CIDR.VOD_BASE_DIRcan no longer be deleted as archives.Fixes
/agband/datenschutzreturned HTTP 500 on fresh installs..corrupt-<ts>.fuser -k./healthzworks on systemd hosts.Infrastructure
compileall,ruff check ., unit tests,bash -nfor shell scripts; Dependabot.Verification
ruff check .: all checks passed.🤖 Generated with Claude Code