Skip to content

feat(detect): NVDEC frame source via ffmpeg pipe with cv2 fallback - #18

Merged
joepetjr merged 2 commits into
mainfrom
feat/nvdec-frame-source
Nov 9, 2026
Merged

joepetjr merged 2 commits into
mainfrom
feat/nvdec-frame-source

Conversation

@joepetjr

@joepetjr joepetjr commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Why

Stage 1 (detect_and_track) is the long pole of every GoMoveShift drive: ~3.2 h for a 48-min 4K60 HEVC clip at 15-20 fps while the GPU sits near 30%. The loop reads every frame through cv2.VideoCapture (CPU decode; the installed cv2 has no CUDA). Measured on this box, 240 frames of a real 4K60 HEVC master while a drive was running:

frame source fps
cv2 read (today) 18.1
ffmpeg NVDEC (-hwaccel cuda -c:v hevc_cuvid) piped bgr24, this PR 31.8

Same shape, same fps and frame-count metadata, same sampled frame indices.

What

  • redactcam/frames.py (new): iter_frames(path, *, decode="auto", start_frame=0, capture=None) -> (FrameMeta, iterator). Metadata still from cv2.VideoCapture. Backends cv2 (unchanged behaviour) and nvdec (ffmpeg pipe; codec from ffprobe → hevc_cuvid/h264_cuvid/av1_cuvid). read_full loops readinto until a whole frame; zero bytes = end of stream; a partial frame or an early end (nonzero exit, or >1% short of the container's count) raises FrameSourceError, never a silent truncation. Close: stdout closed, 10 s wait, terminate, kill.
  • auto (and an explicit nvdec) fall back to cv2 with one INFO line when: ffmpeg missing, no -hwaccel cuda, no *_cuvid decoder for the codec, the 1-frame probe fails, the pipe yields no first frame, the clip carries a rotate tag / display matrix (not proven to apply under cuda), ffprobe and cv2 disagree on frame size, or start_frame > 0 (checkpoint resume; NVDEC path does not seek). REDACTCAM_DECODE=cv2|nvdec|auto overrides.
  • detect.detect_and_track and detect_regions consume frames through it (decode= argument, default auto). Return types unchanged. Chosen backend logged (detect: decode=…) only while progress logging is on.
  • README "Decode backend (NVDEC)" section; CHANGELOG Unreleased › Added.

Tests

tests/test_frames.py (new): cv2/nvdec parity on synthetic h264 and hevc clips (frame_count, shape, dtype, mean abs diff < 2), short-read pipe reader, partial final frame raises, auto fallback when ffmpeg is missing or the probe fails, env override, identical sampled indices in detect_and_track for both backends (stub detector). NVDEC tests skip with a reason where cuda/cuvid is absent; on this box they ran.

  • Red: ImportError for redactcam.frames (.data/logs/nvdec-red.log).
  • Green: 292 passed, 0 skipped (nvdec-green.log); dispatcher re-run unpiped: 292 passed (nvdec-verify.log).
  • ruff check . clean. ruff format --check flags 15 files, 9 untouched by this PR: the venv's ruff differs from the repo's pin; nothing reformatted.
  • No existing test changed. One existing test (test_zero_emits_no_progress_lines) caught the new log line firing with progress off; the code was fixed, not the test.

Not done: seeking for start_frame > 0 on the NVDEC path (ffmpeg -ss accurate seek is a follow-up); rotated clips stay on cv2; coverage.py and timeline.py still open cv2.VideoCapture directly. Diff 611 sum.

CI has neither the face ONNX nor the centerface extra; the logging test
ran detect_and_track with a real detector and failed on ModuleNotFoundError
(deface). The nvdec parity test had the same gap, hidden by its skip.
@joepetjr
joepetjr merged commit ce6a1c2 into main Nov 9, 2026
4 checks passed
@joepetjr
joepetjr deleted the feat/nvdec-frame-source branch November 9, 2026 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant