[STREAM] Video Ingestion & Dynamic Frame Sampling Service

[hdenizkaraman]

🎯 Task Description

apps/stream turns an uploaded video into the smallest set of frames that still contains every safety-relevant event, with accurate timestamps — and exposes that capability to the AI agent as a set of callable tools rather than a fire-and-forget pipeline.

Scope correction (şartname + mentor e-mail)

The competition input model is file upload, not live capture: "Operasyon sahasında bir video sisteme yüklenir." The final is scored on short, previously-unseen test videos given to every team under identical conditions. RTSP / continuous capture is therefore out of scope for the Grand Final and tracked as a post-final stretch goal. This removes back-pressure, frame-drop and buffering concerns from this issue entirely.

The core constraint

VLM context is the bottleneck, and it is the reason this service exists:

2 min video @ 30 fps	3600 frames
Naive 1 fps sampling	120 frames
~800 tokens/frame (to be measured, resolution-dependent)	~96k tokens
Realistic budget for one pass	16–64 frames

So ~120 candidate frames must become ~32 — and the accident will be in exactly the frame we drop. Two failure modes, both fatal:

• Too few frames → the 00:15 rollover is missed → event detection fails → the %35 functionality score is gone.
• Too many frames → context overflows, latency explodes → the performance criteria are gone.

The correct balance differs per video. A fixed frame count or a fixed motion threshold fails on one side or the other, and şartname §4 explicitly states: "Statik, yalnızca kural tabanlı çözümler düşük puanlanacaktır." Adaptivity here is a real engineering requirement, not just a scoring one.

---

🔍 Design: coarse-to-fine temporal zoom

Treat video analysis like zooming on a map, not like looking at a photo.

                    ┌──────────────────────────────────────────┐
  uploaded video ──►│ PASS 1 — motion profile      (CPU, ~sec) │
                    │ decode @160x90 gray, frame differencing  │
                    │ ⇒ 1-D "where something happens" curve    │
                    └────────────────┬─────────────────────────┘
                                     ▼
                    ┌──────────────────────────────────────────┐
                    │ PASS 2 — coarse look        (~16 frames) │
                    │ adaptive sampling weighted by motion     │
                    │ ⇒ "what is going on in this video?"      │
                    └────────────────┬─────────────────────────┘
                                     ▼
                    ┌──────────────────────────────────────────┐
                    │ PASS 3 — zoom          (agent-triggered) │
                    │ agent: "something at ~00:14, zoom in"    │
                    │ ⇒ 12 dense frames from 00:12–00:17       │
                    │ ⇒ "what exactly happened, and at when?"  │
                    └──────────────────────────────────────────┘

Total frames sent to the VLM: ~28 instead of 120, with the critical moment localized precisely.

Pass 1 — motion profile

Decode the whole video downscaled to 160x90 grayscale and compute per-frame SAD (sum of absolute differences) against the previous frame. No GPU, no OpenCV, thousands of frames/sec. Output: [(timestamp, motion_score)] plus scene-cut positions.

Pass 2 — adaptive selection

Not "1 frame per second" and not "threshold > X" — both are static. Instead, sample uniformly in motion-space rather than time-space:

1. Build the cumulative motion curve M(t) = Σ m(t).
2. Place N points equally spaced along the M axis and map them back to t.
3. Dense motion regions automatically receive dense sampling; static regions receive sparse sampling. No threshold is ever chosen by hand.

Guard rails:

• Scene-cut boundaries are always included.
• max_gap guarantees at least one frame per quiet stretch (a static frame is still evidence — e.g. a motionless person on the ground).
• min_gap + pHash dedup drops near-identical frames.

Pass 3 — zoom, as a tool

Pass 3 is deliberately not automated. It is exposed to the agent, which decides when and where to zoom.

This is a scoring decision as much as an architectural one:

• şartname §7 — "Mock fonksiyonların ajanın araçları olarak başarıyla kullanılması" (part of %35) — satisfied directly.
• şartname §7 — "Otonomi ve Zeka" (%20): dinamik araç seçimi, çok adımlı karar zincirleri — satisfied directly.
• It gives the jury a one-line story: our agent does not watch the video, it investigates it.

Timestamp grounding

VLMs are weak at "when" — given 16 frames they know the order, not the clock. But the şartname scores precisely on timestamps. Two cheap mitigations, both to be A/B measured:

1. Burn the timestamp into the frame (rendered overlay, corner).
2. Pass a parallel text index: "Frame 7 → t=00:15.2".

---

🛠️ Affected Domain / Package

• apps/dashboard (Next.js Live Monitoring Panel)
• apps/identity (Ory Kratos/Keto Identity & Authorization)
• apps/gateway (Rust Axum API Gateway & Realtime SSE)
• apps/stream (Video Ingestion & Dynamic Frame Sampling)
• apps/ai (Agentic Reasoning & Decision Logic)
• packages/optics (Image & Media Processing Toolkit)
• packages/database (SurrealDB, Qdrant)
• platform/ or tools/ (Docker, Infrastructure & Benchmarks)
• Other: packages/event-sdk (shared message contracts)

---

🏗️ Technical Decisions

Decision	Choice	Rationale
Media I/O	ffmpeg / ffprobe as subprocesses, raw frames over a pipe	Avoids OpenCV Rust binding builds entirely (a real time sink on Windows). Only requires ffmpeg on PATH — one documentation line.
Pixel math	Hand-written in Rust over raw grayscale buffers	Frame differencing is ~15 lines. No dependency, and it preserves the measurable Rust throughput story the şartname's performance criteria reward.
Motion signal	Frame differencing, not dense optical flow	We need how much changed, not which direction. Direction is what the VLM interprets. See #6.
pHash	Dedup only	Cheap removal of near-identical frames on a static CCTV shot. Not used for selection.
Frame delivery	MinIO object keys, not base64 in the message	Keeps NATS payloads small; consumers fetch what they need.
Analysis vs. delivery resolution	160x90 gray for analysis, full quality for the VLM	Standard split: cheap to decide, high fidelity to send.

---

🔌 Tool API (agent-facing contract)

The surface the orchestrator may call. This is the contract other issues depend on — changes here need a heads-up in #6.

video_info(video_id)                          -> { duration_ms, fps, width, height, size_bytes }
motion_profile(video_id, bucket_ms)           -> [{ t_ms, score, is_scene_cut }]
sample_overview(video_id, budget)             -> [{ t_ms, frame_uri, motion_score }]
zoom_range(video_id, t0_ms, t1_ms, budget)    -> [{ t_ms, frame_uri, motion_score }]
get_frame(video_id, t_ms, max_dim)            -> { t_ms, frame_uri }
crop_region(video_id, t_ms, bbox)             -> { t_ms, frame_uri }

All frames carry their true source timestamp; the agent never has to infer time from ordering.

---

📡 NATS Contracts (draft — to be finalised in packages/event-sdk)

stream.video.ingested

{ "video_id": "...", "object_key": "raw/<id>.mp4", "duration_ms": 122000,
  "fps": 30.0, "width": 1920, "height": 1080, "ingested_at": "..." }

stream.frame.extracted

{ "video_id": "...", "batch_id": "...", "pass": "overview|zoom",
  "frames": [ { "t_ms": 15200, "object_key": "frames/<id>/015200.jpg",
                "motion_score": 0.87, "is_scene_cut": false } ] }

---

📋 Task Checklist

Ingestion

• Accept video upload, validate container/codec, persist raw blob to MinIO
• Probe metadata via ffprobe (duration, fps, resolution, rotation)
• Publish stream.video.ingested

Pass 1 — motion profile (packages/optics)

• Stream-decode to 160x90 grayscale via ffmpeg pipe
• Per-frame SAD against previous frame; normalise to 0..1
• Scene-cut detection from the motion curve
• Per-frame pHash for dedup
• Persist the profile so passes 2/3 never recompute it

Pass 2 — adaptive selection

• Cumulative-motion inverse-transform sampling with N budget
• min_gap / max_gap guard rails + scene-cut forcing
• pHash dedup on the selected set
• Full-quality extraction of selected frames via a second ffmpeg call
• Optional timestamp overlay renderer (feature-flagged for A/B)
• Publish stream.frame.extracted

Pass 3 — zoom + tool surface

• Implement zoom_range, get_frame, crop_region, motion_profile, video_info
• Expose the tool surface to the orchestrator (transport per [PIPELINE] Smart Pre-Processing & Pre-Prompt Feature Extraction #6 decision)
• Cache decoded segments so repeated zooms into one region stay cheap

Validation

• Bench harness over the Golden Dataset ([DATASET] Golden Dataset Curation & Benchmark Ground Truth #5) reporting the KPIs below
• A/B: timestamp overlay on vs. off
• A/B: adaptive sampling vs. uniform 1 fps baseline

---

📊 KPIs (feeds the mandatory measurement report, şartname §4/§6)

Metric	Definition	Target
Event coverage recall	% of Golden Dataset ground-truth events with ≥1 selected frame within ±1.0 s	≥ 95%
Frame reduction ratio	selected frames ÷ total frames	≥ 100×
Timestamp error	|reported − ground truth| for detected events	< 1.0 s
Pass-1 throughput	analysis frames/sec, and × realtime	measure & report
Preprocessing latency	wall-clock seconds per minute of video	measure & report
Peak RSS	resident memory during pass 1	must stay bounded and independent of video length

The first three are properties of the algorithm and hold on any machine, so they carry hard targets. The last three depend on the final hardware, which is not yet fixed — they are measured and reported as şartname §4 requires. Crucially, the adaptive-vs-uniform baseline comparison is run on identical hardware, so the delta stays meaningful regardless of which machine we end up with.

Event coverage recall is the metric that matters most: it isolates "did sampling lose the event?" from "did the VLM understand it?" — a stream-side failure we can measure and fix without touching the model. It requires #5.

---

✅ Definition of Done

• A video file in → stream.frame.extracted out, end to end, over NATS
• All six tools callable by the orchestrator and covered by the shared contract
• KPI table above filled with real numbers on the Golden Dataset
• Adaptive sampling beats the uniform-1fps baseline on event coverage recall at equal frame budget
• ffmpeg dependency + run steps documented in documents/features/stream-service.md

---

🚫 Out of Scope

• Live RTSP / continuous capture (post-final stretch goal)
• YOLO bounding boxes and CLIP embeddings — measured and decided in [PIPELINE] Smart Pre-Processing & Pre-Prompt Feature Extraction #6, not assumed here
• Any fixed rule that decides whether an event occurred; this service decides where to look, the model decides what happened

---

🔗 Related Resources / Docs

• Depends on: [PIPELINE] Smart Pre-Processing & Pre-Prompt Feature Extraction #6 (pipeline decision), [DATASET] Golden Dataset Curation & Benchmark Ground Truth #5 (Golden Dataset, for the recall metric)
• Blocks: [GATEWAY] Rust Axum API Gateway & Realtime SSE Streaming #3 (gateway consumes event.risk.detected downstream of this)
• Şartname: 3. Senaryo §3 Problem Tanımı, §4 Temel Beklentiler, §7 Değerlendirme Kriterleri
• Internal Docs: documents/features/stream-service.md