Skip to content

Persist video assets and add bounded cancellable processing jobs - #4

Merged
Artsen merged 3 commits into
mainfrom
feat/persistent-video-jobs
Jul 26, 2026
Merged

Persist video assets and add bounded cancellable processing jobs#4
Artsen merged 3 commits into
mainfrom
feat/persistent-video-jobs

Conversation

@Artsen

@Artsen Artsen commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Summary

  • Migrates validated manifest v1 image records to strict manifest v2 with private pre-migration recovery copies.
  • Adds persistent video sources, outputs, and inspectable job history.
  • Adds a bounded FIFO worker manager with cooperative cancellation and restart recovery.
  • Preserves the first source audio stream by default and produces H.264/yuv420p/fast-start MP4 with AAC audio.
  • Adds deterministic timestamp previews and reusable ID-based single-range streaming.
  • Moves the current Flask UI from jQuery/legacy filenames to the persistent APIs while retaining deprecated compatibility routes.

Storage contract

  • New strict discriminated VideoSourceRecord, VideoOutputRecord, and VideoJobRecord types share the v2 sources, outputs, and jobs manifest collections with existing image records.
  • Managed files live under sources/videos, outputs/videos, and temporary/video-jobs; only normalized relative paths are persisted.
  • V1 is detected before validation, validated with ManifestDocumentV1, copied privately, atomically migrated, and never rewritten when invalid.
  • Backup-v1 recovery, second-restart idempotence, media-specific orphan metrics, missing-media reconciliation, leases, cascade deletion, quarantine rollback, and temporary-job cleanup are covered.
  • MediaAssetRepository is canonical; ImageAssetRepository remains a compatibility alias.

Job contract

  • States: queued ? preparing ? processing ? muxing ? completed, with controlled failed and canceled terminals.
  • Default capacity is four queued jobs with one daemon worker; saturation returns HTTP 429.
  • Progress is stage-based and manifest writes are throttled instead of occurring per frame.
  • Queued cancellation is immediate; processing checks between frames; muxing terminates and then kills FFmpeg when needed.
  • Interrupted jobs remove temporary artifacts, increment attempts, set recoveredAfterRestart, and requeue below the two-attempt limit. Missing media and exhausted attempts fail cleanly.
  • Only terminal job history can be deleted; deleting history does not delete output media.

Media contract

  • Admission accepts MP4, MOV, MKV, and AVI through multipart field video, validates decoded FFprobe content with a timeout, rejects extension/container mismatches, nonzero rotation, invalid rates, corrupt media, and configured duration/pixel limits.
  • Timestamp previews return temporary no-store PNG bytes and expose the actual zero-based frame index.
  • Persistent output is always .mp4, video/mp4, H.264, yuv420p, and fast-start. Preserve mode maps only the first audio stream to AAC at 192 kbps; remove mode emits no audio.
  • Final output is probed and profile-validated before output creation and job completion are committed together.

API contract

  • Sources: list, metadata, original, download, timestamp preview, job submission, and delete/cascade.
  • Jobs: list/state filter, metadata, cancel, and terminal-history delete.
  • Outputs: list, inline stream, metadata, download, and delete.
  • Full GET, HEAD, closed, open-ended, and suffix ranges stream in bounded chunks; invalid, multiple, and unsatisfiable ranges return 416 with the complete size. Leases remain held until response close.
  • /upload_preview, /process_video_async, /progress/<task_id>, /video/<filename>, and /download/<filename> remain deprecated compatibility routes. The active UI no longer calls them.

Craft discovery

  • Version is now 0.2.0; manifest/storage schema is 2; video-job contract is 1; Recipe remains v1.
  • Adds persistent-video-library, bounded-video-jobs, video-job-cancellation, timestamp-video-preview, audio-preserving-video-export, and range-streaming capabilities.
  • Metadata/readiness/storage now include video source/output/job counts, jobs by state, queue capacity/active work, job schema, output profile, audio modes, separated media bytes, temporary-job bytes, tool readiness, and reconciliation state.

Validation

  • python check.py: passed Ruff format, Ruff lint, strict Mypy, 197 Python tests, and repository consistency.
  • Python statement coverage: 96.74% (2550/2636).
  • Python branch coverage: 90.08% (463/514); combined coverage is 95.65%.
  • npm test: 7 Playwright tests passed, including the existing image regressions plus video upload/metadata/timestamp preview/job completion/download and accessible cancellation.
  • Axe: zero serious or critical findings in the covered image and video states.
  • Real FFmpeg/FFprobe integration: generated tiny media at runtime; timestamp preview and full processing passed; preserve mode produced H.264/yuv420p + AAC, and remove mode produced no audio.
  • Range coverage: full, HEAD, closed, open-ended, suffix, invalid, multiple, and 416 behavior passed.
  • Restart/cancellation smoke: queued, interrupted, muxing/completion reconciliation, attempt limit, missing media, queued/processing cancellation, and partial cleanup passed.
  • git diff --check: passed.
  • GitHub Actions: passed (quality and browser).

Known limitations

  • One processing worker by default and one application writer process per data root.
  • OpenCV processing may normalize variable-frame-rate sources to the probed constant rate.
  • Only the first audio stream is preserved; subtitles, chapters, attachments, and additional audio tracks are not preserved.
  • Nonzero display rotation is rejected rather than normalized.
  • No hardware acceleration or detailed delivery encoding controls; those remain Web Video Optimizer responsibilities.
  • Legacy endpoints remain temporarily; the frontend is still transitional Flask/JavaScript.
  • React, the frontend/API split, Craft dashboard control, and WVO handoff are deferred.
  • No authentication exists; the application remains loopback-only and must not be exposed directly to a public network.

@Artsen
Artsen marked this pull request as ready for review July 26, 2026 05:55
@Artsen
Artsen merged commit bdef597 into main Jul 26, 2026
2 checks passed
@Artsen
Artsen deleted the feat/persistent-video-jobs branch July 26, 2026 05:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant