From cdeeb17d081665f73135b065e038ae23deb27900 Mon Sep 17 00:00:00 2001 From: "user.email" Date: Sat, 21 Mar 2026 15:33:55 +0530 Subject: [PATCH] Update README with GPU acceleration docs and v2.0.0 changes Fixes #19 --- README.md | 208 ++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 187 insertions(+), 21 deletions(-) diff --git a/README.md b/README.md index 001c1f7..07162fd 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,8 @@ Phase-based motion magnification amplifies subtle motions invisible to the naked eye. Unlike Eulerian (color-based) methods that amplify pixel intensity changes, phase-based magnification operates on the phase of complex wavelet coefficients — which directly encode local position — enabling 10–100x amplification with fewer artifacts. This is a Python implementation based on [Wadhwa et al. (SIGGRAPH 2013)](https://people.csail.mit.edu/nwadhwa/phase-video/) using the 2D Dual-Tree Complex Wavelet Transform. +**v2.0.0** adds GPU acceleration via PyTorch, delivering ~5x end-to-end speedup on CUDA-capable GPUs. + --- ## Table of Contents @@ -27,14 +29,21 @@ Phase-based motion magnification amplifies subtle motions invisible to the naked - [Phase Extraction](#phase-extraction) - [Temporal Filtering](#temporal-filtering) - [Phase Modification and Reconstruction](#phase-modification-and-reconstruction) +- [GPU Acceleration](#gpu-acceleration) + - [Two-Pass Batched Architecture](#two-pass-batched-architecture) + - [Chunked cuFFT Temporal Filtering](#chunked-cufft-temporal-filtering) + - [Design Decisions](#design-decisions) + - [Performance](#performance) - [Setup](#setup) - [A. Google Colab](#a-google-colab) - [B. Local Setup](#b-local-setup) - - [C. Docker](#c-docker) + - [C. Docker (CPU)](#c-docker-cpu) + - [D. Docker (GPU)](#d-docker-gpu) - [Usage](#usage) - [CLI Tool](#cli-tool) - [Notebook](#notebook) - [Tips](#tips) +- [v2.0.0 Breaking Changes](#v200-breaking-changes) - [Development](#development) - [Running Tests](#running-tests) - [Versioning](#versioning) @@ -80,13 +89,37 @@ The DTCWT is ~5x faster than complex steerable pyramids while still providing re The algorithm has five stages: +``` +Input Video + | + v +[1. Forward 2D DTCWT] ──> Complex coefficients C = A * e^(i*phi) + | (nlevels scales x 6 orientations per frame) + v +[2. Phase Extraction] ──> Cumulative phase phi(t) via frame-to-frame + | complex division + cumsum + v +[3. Temporal Filter] ──> Separate base motion phi_0 (slow) + | from detail motion (phi - phi_0) + v +[4. Phase Modification] ──> Amplify detail: phi_0 + (phi - phi_0) * k + | + smoothing pass (width=2) + v +[5. Inverse DTCWT] ──> Reconstruct with |C| * e^(i*phi_modified) + | + v +Output Video (magnified motions) +``` + +Each color channel (R, G, B) is processed independently through the full pipeline, then recombined for the output video. + **1. Forward 2D DTCWT** Each video frame is decomposed into `nlevels` scales × 6 orientations, producing complex coefficients $C(s, \theta, x, y, t) = A \cdot e^{i\phi}$ where amplitude $A$ encodes texture strength and phase $\phi$ encodes spatial position. **2. Phase Extraction** -Cumulative phase is computed via frame-to-frame complex division. For each coefficient, dividing frame $t$'s normalized value by frame $t-1$'s gives the phase ratio. Taking `angle()` and `cumsum()` produces $\phi(t)$ — the absolute phase relative to frame 0. Complex division is more numerically stable than direct phase subtraction. +Cumulative phase is computed via frame-to-frame complex division. For each coefficient, dividing frame $t$'s normalized value by frame $t-1$'s gives the phase ratio. Taking `angle()` and `cumsum()` produces $\phi(t)$ — the absolute phase relative to frame 0. Complex division is more numerically stable than direct phase subtraction because it naturally handles phase wrapping at $\pm\pi$ boundaries. **3. Temporal Filtering** @@ -117,7 +150,7 @@ Coefficients are reconstructed with the original amplitude and modified phase: $ - **Higher k → more noise/artifacts** — amplification also amplifies phase noise, producing spatial artifacts at high magnification factors. - **Memory intensive** — all frame pyramids must remain in memory simultaneously for temporal filtering. Long videos or high resolutions may require significant RAM. -- **Slow** — DTCWT is computed on every frame × 3 color channels. Processing time scales linearly with frame count. +- **Slow on CPU** — DTCWT is computed on every frame × 3 color channels. Processing time scales linearly with frame count. Use `--gpu` for ~5x speedup. - **Large motions violate assumptions** — the phase-to-motion relationship is linear only for small displacements. Large motions produce phase wrapping artifacts. --- @@ -134,13 +167,82 @@ The `normalize_phase()` function normalizes complex coefficients to unit magnitu `flattop_filter_1d()` applies a flat-top window (from `scipy.signal.windows.flattop`) as a low-pass smoothing kernel along the time axis. The window size is `width / 0.2327`, where 0.2327 is the flat-top window's equivalent noise bandwidth in bins. This filter separates the slow baseline motion from the fast detail motion we want to amplify. +For windows larger than 32 samples, the filter switches to FFT-based convolution (`scipy.signal.fftconvolve`) for a ~4x speedup. + ### Phase Modification and Reconstruction After filtering, the baseline phase $\phi_0$ is subtracted from the total phase to isolate detail motion. This detail is multiplied by the magnification factor $k$, then added back: $\phi_0 + (\phi - \phi_0) \times k$. An additional smoothing pass with width=2 removes high-frequency phase noise that would appear as spatial flickering. The final coefficients are reconstructed by preserving the original amplitude and applying the modified phase: $|h| \cdot e^{i\hat{\phi}}$. -Each color channel (R, G, B) is processed independently through the full pipeline, then recombined for the output video. +--- + +## GPU Acceleration + +The `--gpu` flag enables GPU-accelerated processing via [PyTorch](https://pytorch.org/) and [`pytorch_wavelets`](https://github.com/fbcotter/pytorch_wavelets). The GPU path replaces both the DTCWT transforms and temporal filtering with CUDA-accelerated equivalents while keeping the same algorithmic pipeline. + +### Two-Pass Batched Architecture + +Storing all DTCWT coefficients (amplitudes + phases) for every frame would require ~718 MB of CPU RAM per channel. The GPU path avoids this with a two-pass design: + +**Pass 1 — Forward DTCWT + Phase Extraction:** +- Frames are sent to the GPU in batches (batch size auto-tuned to ~70% of available VRAM) +- Forward DTCWT produces complex coefficients; only the **phase** is extracted and stored on CPU +- Amplitudes and lowpass coefficients (Yl) are **discarded** — they will be recomputed in Pass 2 +- Cross-batch boundary handling carries the last frame's normalized coefficients to the next batch, ensuring bitwise-identical results to single-batch processing + +**Temporal Filtering on GPU** (between passes): +- Phase arrays are filtered using cuFFT (see below) + +**Pass 2 — Reconstruction + Inverse DTCWT:** +- Forward DTCWT is re-run on the original frames to recover amplitudes and Yl (deterministic — verified 0.00 diff between runs) +- Modified phases from the filtered output are combined with recovered amplitudes: `real = amp * cos(phase)`, `imag = amp * sin(phase)` +- Inverse DTCWT produces the output frames + +### Chunked cuFFT Temporal Filtering + +The temporal filter is the pipeline bottleneck (54% of CPU runtime). On GPU, it uses `torch.fft` (backed by cuFFT): + +1. Phase arrays are chunked along the coefficient dimension to fit in VRAM — cuFFT requires ~20x the array size in working memory, so a 538 MB phase array would need ~3.8 GB for a whole-array FFT +2. Each chunk is transferred to GPU, FFT'd along the time axis, multiplied by the pre-computed FFT of the flat-top window, then inverse FFT'd +3. The magnification and smoothing passes are applied on-GPU before transferring back + +**Chunk size auto-tuning:** queries `torch.cuda.mem_get_info()` and uses 70% of free VRAM as the limit, adapting to any GPU without user configuration. + +**Boundary handling:** The GPU path uses zero-padding (not reflect-padding like CPU) for FFT convolution. This produces ~1.3% relative error at the first and last few frames, which at 65+ dB PSNR is visually imperceptible. + +### Design Decisions + +| Decision | Choice | Why | +|---|---|---| +| Two-pass vs store amplitudes | Two-pass (recompute) | Saves ~718 MB RAM/channel; forward DTCWT is deterministic, adds <1s on GPU | +| C=1 sequential vs C=3 batched | 3x C=1 sequential | C=3 only speeds DTCWT (22% of pipeline) by 2.2x but costs 2.5x VRAM; 1.2x total speedup not worth the complexity | +| Float32 vs float64 | Float32 everywhere | PyTorch/CUDA standard; cumsum error max 2.4e-4 rad at 900 frames, 1000x below visibility threshold | +| Whole-array vs chunked cuFFT | Chunked | Whole-array OOMs on consumer GPUs (>100 frames at 528x592); chunking is still 3x faster than CPU FFT | +| Zero-pad vs reflect-pad (GPU FFT) | Zero-pad | Reflect-padding would increase memory; 1.3% boundary error at 65+ dB PSNR is visually imperceptible | + +See [`docs/design/gpu-acceleration.md`](docs/design/gpu-acceleration.md) for the full design document including alternatives considered and tradeoff analysis. + +### Performance + +Benchmarked on face.mp4 (301 frames, 528x592, k=3) with an RTX 4050 (6 GB VRAM): + +| Metric | CPU | GPU | +|---|---|---| +| Per-channel speedup | — | ~5-17x | +| End-to-end time | ~2 min | ~24 sec | +| Precision | float64 | float32 | +| Peak RAM | ~1.2 GB | ~800 MB | +| Peak VRAM | — | ~2-3 GB | + +**Hardware requirements (GPU path):** +- NVIDIA GPU with CUDA 12.1+ support +- Minimum ~4 GB VRAM recommended (auto-tuning adapts batch/chunk sizes) +- [`nvidia-container-toolkit`](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) for Docker GPU support + +**Pre-flight memory check:** Before processing, the tool estimates peak CPU RAM and VRAM usage and warns if it may exceed available resources, with suggestions to reduce `--nlevels`, resolution, or switch to CPU mode. + +**Note:** CPU and GPU paths produce different outputs — they use different DTCWT implementations (`dtcwt` vs `pytorch_wavelets`) at different precisions (float64 vs float32). Both produce valid motion magnification results; they are not cross-comparable. --- @@ -170,7 +272,7 @@ jupyter notebook MotionMagDtcwt.ipynb **Requirements:** Python 3.8+ -### C. Docker +### C. Docker (CPU) ```bash # Build @@ -179,10 +281,27 @@ jupyter notebook MotionMagDtcwt.ipynb # Run docker run --rm -it \ -v "$(pwd)":/app/data \ - motion-magnification-dtcwt \ + motion-mag-dtcwt:latest \ + -i /app/data/input.mp4 -o /app/data/output.avi +``` + +### D. Docker (GPU) + +Requires [nvidia-container-toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html). + +```bash +# Build +./docker-build-gpu.sh + +# Run +docker run --rm -it --gpus all \ + -v "$(pwd)":/app/data \ + motion-mag-dtcwt-gpu:latest \ -i /app/data/input.mp4 -o /app/data/output.avi ``` +The GPU Docker image is based on `pytorch/pytorch:2.1.2-cuda12.1-cudnn8-runtime` and includes PyTorch, `pytorch_wavelets`, and all dependencies. The `--gpu` flag is the default entrypoint behavior in the GPU image. + --- ## Usage @@ -190,9 +309,15 @@ docker run --rm -it \ ### CLI Tool ```bash +# CPU (default) python motion_mag.py -i face.mp4 python motion_mag.py -i face.mp4 -o magnified.avi -k 5 python motion_mag.py -i face.mp4 -k 3 -w 80 --nlevels 6 + +# GPU +python motion_mag.py -i face.mp4 --gpu +python motion_mag.py -i face.mp4 --gpu --device 1 -k 10 +python motion_mag.py -i face.mp4 --gpu -k 5 --biort near_sym_a --qshift qshift_a ``` | Flag | Default | Description | @@ -202,8 +327,16 @@ python motion_mag.py -i face.mp4 -k 3 -w 80 --nlevels 6 | `-k / --magnification` | 3 | Magnification factor | | `-w / --width` | 80 | Temporal filter width (frames) | | `--nlevels` | 8 | DTCWT decomposition levels | +| `--gpu` | off | Enable GPU acceleration (requires PyTorch + pytorch_wavelets) | +| `--device` | 0 | CUDA device index (for multi-GPU systems) | +| `--biort` | `near_sym_b` | Biorthogonal wavelet filter for DTCWT level 1 | +| `--qshift` | `qshift_b` | Quarter-shift wavelet filter for DTCWT levels 2+ | | `--version` | — | Show program version and exit | +**Available wavelet filters:** +- `--biort`: `antonini`, `legall`, `near_sym_a`, `near_sym_b` +- `--qshift`: `qshift_06`, `qshift_a`, `qshift_b`, `qshift_c`, `qshift_d` + ### Notebook Open the notebook and run all cells. By default, it downloads a sample face video from the original paper and magnifies it. To use your own video, change the `filename` variable. @@ -214,6 +347,19 @@ Open the notebook and run all cells. By default, it downloads a sample face vide - Larger filter width → smoother temporal filtering, better for slow motions (breathing, pulse). - Fewer `nlevels` → faster processing but less spatial detail captured. - R, G, B channels are processed independently — color artifacts indicate magnification is too high. +- Use `--gpu` for ~5x faster processing if you have an NVIDIA GPU. + +--- + +## v2.0.0 Breaking Changes + +- **Default wavelet filters changed** from `near_sym_a`/`qshift_a` to `near_sym_b`/`qshift_b`. The longer `near_sym_b` filters produce fewer block artifacts at higher magnification factors (k=5+). To restore v1.x behavior: + ```bash + python motion_mag.py -i input.mp4 --biort near_sym_a --qshift qshift_a + ``` +- **Output differs from v1.x** even on CPU due to the filter change. Pin filters explicitly if reproducibility with older versions is needed. + +See [CHANGELOG.md](CHANGELOG.md) for full release history. --- @@ -224,14 +370,18 @@ Open the notebook and run all cells. By default, it downloads a sample face vide All tests run inside Docker — no local Python dependencies needed: ```bash -# Run lint + unit tests (builds image automatically if not found) +# CPU: lint + unit tests (builds image automatically if not found) ./test.sh +# GPU: lint + unit tests including CUDA tests (requires nvidia-container-toolkit) +./test.sh gpu + # Force rebuild before testing ./test.sh --build +./test.sh gpu --build ``` -**Tests** (`tests/test_motion_mag.py`) cover: +**CPU tests** (`tests/test_motion_mag.py`) cover: - Phase normalization (unit magnitude, zero safety) - Flat-top temporal filter (DC passthrough, smoothing, edge cases) - Temporal phase extraction (constant phase, output shape) @@ -239,9 +389,18 @@ All tests run inside Docker — no local Python dependencies needed: - `load_video` buffer safety - All CLI input validation error paths +**GPU tests** (`tests/test_motion_mag_gpu.py`) cover: +- GPU forward/inverse DTCWT roundtrip +- Phase extraction (finite values, correct shapes) +- Batched vs single-batch consistency (cross-batch boundary verification) +- cuFFT temporal filter (shape preservation, DC signal handling) +- Full GPU pipeline smoke test (finite output, correct dimensions) +- Memory estimation arithmetic +- All GPU tests skip automatically on systems without CUDA + **Dev workflow:** 1. Make your changes -2. Run `./test.sh` +2. Run `./test.sh` (and `./test.sh gpu` if touching GPU code) 3. If all tests pass, commit and open a PR 4. CI runs lint + smoke tests automatically @@ -256,23 +415,30 @@ Version is tracked in a `VERSION` file at the project root. `motion_mag.py` has 4. Commit: `Release vX.Y.Z` 5. Tag: `git tag -a vX.Y.Z -m "Release vX.Y.Z"` 6. Push: `git push && git push origin vX.Y.Z` -7. Rebuild Docker image: `./docker-build.sh` +7. Rebuild Docker images: `./docker-build.sh && ./docker-build-gpu.sh` ### Project Structure ``` -motion_mag.py # CLI tool -MotionMagDtcwt.ipynb # Jupyter notebook -Dockerfile # Docker image -docker-build.sh # Build + tag image -test.sh # Run lint + unit tests (Docker) -requirements.txt # Runtime dependencies -requirements-dev.txt # Dev dependencies (pytest, ruff) +motion_mag.py # CLI tool (CPU + GPU paths) +MotionMagDtcwt.ipynb # Jupyter notebook +Dockerfile # CPU Docker image (python:3.11-slim) +Dockerfile.gpu # GPU Docker image (pytorch:2.1.2-cuda12.1) +docker-build.sh # Build + tag CPU image +docker-build-gpu.sh # Build + tag GPU image +test.sh # Run lint + tests (Docker, supports cpu/gpu mode) +requirements.txt # CPU runtime dependencies +requirements-gpu.txt # GPU runtime dependencies +requirements-dev.txt # Dev dependencies (pytest, ruff) tests/ - test_motion_mag.py # Unit tests -docs/design/ # Architecture decision records -VERSION # Single source of truth for version -CHANGELOG.md # Release history + test_motion_mag.py # CPU unit tests + test_motion_mag_gpu.py # GPU unit tests (CUDA-only, skip on CPU) +docs/design/ # Architecture decision records + gpu-acceleration.md # GPU design doc + dtcwt-hardening.md # Hardening design doc +VERSION # Single source of truth for version +CHANGELOG.md # Release history +CONTRIBUTING.md # Contribution guidelines ``` ---