[PIPELINE] Smart Pre-Processing & Pre-Prompt Feature Extraction

[hdenizkaraman]

🧾 ARCHITECTURAL PROPOSAL (RFC) — updated with a concrete recommendation

The original proposal below has been narrowed into a specific recommendation after reading the şartname. The recommendation is not final — the four design questions are answered with reasoning, and each answer is falsifiable by the benchmark plan at the bottom. Please react in the comments; silence for the next 48h will be read as agreement so the work in #1 can start.

🎯 Task Description

Define the pre-processing chain that sits between raw video and the VLM: what we extract, what we throw away, and what we put into the prompt — with the goal of maximising reasoning accuracy per token of VLM context.

What changed since this issue was opened

Two şartname findings reshape the answer:

1. Input is an uploaded file, not a live stream ("Operasyon sahasında bir video sisteme yüklenir"), and the final is judged on short unseen test videos. Throughput on hour-long streams is not what we are optimising for.
2. §4: "Statik, yalnızca kural tabanlı çözümler düşük puanlanacaktır." Any stage that makes a decision by fixed threshold is a scoring liability. Stages that supply evidence to a model are not.

That distinction — evidence vs. decision — is the principle behind every answer below. Our pipeline should decide where to look; only the model decides what happened.

---

✅ Recommendation

Stage	Decision	Rationale
Frame differencing (160x90 gray)	Core — in	Gives the "where something happens" curve that drives adaptive sampling. CPU-only, ~ms.
Perceptual hashing (pHash)	Core — in, dedup only	Static CCTV footage is largely identical frames. Cheap, high value. Not used to select.
Adaptive sampling + coarse-to-fine zoom	Core — in	The actual answer to the context-budget problem, and the §4 "dynamic" requirement. Detailed in #1.
Timestamp grounding (overlay + text index)	Core — in	Directly serves the scored "kritik anları zaman bilgisi ile belirleme" requirement.
Dense optical flow	Out	We need how much changed, not which direction. Direction is exactly what the VLM interprets for us. Strictly more expensive for information we discard.
YOLOv11x bounding boxes	Deferred — benchmark, then decide	Gives spatial localisation; the şartname scores temporal localisation. The VLM already reports "there is a forklift". Keep the module pluggable, measure the delta, report the result.
CLIP embeddings	Out for the Grand Final	Its value is prefiltering long videos; our test videos are short. It would compete with the VLM for VRAM. Poor complexity/benefit ratio under the current deadline.

---

💬 Answers to the Open Design Questions

1. Is pHash sufficient alone, or do we strictly need Optical Flow?
Neither, as framed. Frame differencing is the right middle. pHash answers a binary ("is this the same frame?"), which cannot modulate sampling density. Optical flow answers more than we need at several times the cost. Frame differencing yields a continuous 0..1 signal — exactly what inverse-transform sampling consumes. Keep pHash, but only as the dedup pass.

2. Do YOLO boxes improve VLM localisation enough to justify the overhead?
Probably not for this competition, and here is the falsifiable claim: YOLO improves spatial grounding, while §3/§4 award points for temporal grounding ("kritik anları zaman bilgisi ile"). Our scarce resources buy more accuracy in temporal zoom than in bounding boxes. Additionally it is a second GPU model competing with the VLM for memory. Action: implement behind a feature flag, measure, and publish the delta — a documented negative result strengthens the report the şartname requires.

3. CLIP prefilter, or route directly to VLM dynamic sampling?
Direct to VLM. CLIP's payoff is triaging thousands of candidate frames; after pass 1 we already have ~120 candidates and a budget of ~32. There is not enough left to triage. Revisit only if we pursue live/long-form video after the final.

4. Synchronous inside apps/stream, or decoupled via NATS?
Decoupled — but the reason is not performance, it is agency. For the agent to call zoom_range(...), apps/stream must be a request/response service, not a fire-and-forget pipeline. Passes 1–2 stay event-driven over NATS (stream.video.ingested → stream.frame.extracted); pass 3 is a synchronous tool call from the orchestrator. If stream were purely a pipeline stage, temporal zoom would be impossible and we would lose the strongest part of the design.

---

🔄 Resulting Pipeline

Raw Video
   │
   ├──▶ ffprobe metadata
   ├──▶ [PASS 1] frame differencing @160x90 gray  ──▶ motion curve + scene cuts
   ├──▶ pHash                                     ──▶ dedup near-identical frames
   │
   ▼
[PASS 2] adaptive sampling (uniform in motion-space, not time-space)
   │        + timestamp grounding
   ▼
   VLM ──── "something around 00:14?" ────┐
   ▲                                      │
   └──── [PASS 3] zoom_range(t0,t1,n) ◀───┘   (agent-invoked tool)

   ( YOLO / CLIP: pluggable, benchmarked, off by default )

---

🛠️ Affected Domain / Package

• apps/dashboard (Next.js Live Monitoring Panel)
• apps/identity (Ory Kratos/Keto Identity & Authorization)
• apps/gateway (Rust Axum API Gateway & Realtime SSE)
• apps/stream (Video Ingestion & Dynamic Frame Sampling)
• apps/ai (Agentic Reasoning & Decision Logic)
• packages/optics (Image & Media Processing Toolkit)
• packages/database (SurrealDB, Qdrant)
• platform/ or tools/ (Docker, Infrastructure & Benchmarks)
• Other (please specify): ``

---

❓ Still Open — team input wanted

• Transport for the pass-3 tool call. apps/ai is Rust (decided), so packages/event-sdk can be a shared crate of serde types rather than a cross-language schema — no codegen needed. Remaining question is only the transport: NATS request/reply (reuses the broker we already run) vs. gRPC/HTTP (simpler to debug). Recommendation: NATS request/reply, since the infrastructure is already there.
• vLLM boundary. vLLM itself is Python-served. Confirm that apps/ai/inference is a thin Rust client against vLLM's OpenAI-compatible HTTP endpoint, rather than an in-process binding — this keeps the whole orchestrator in Rust while still satisfying the şartname's explicit vLLM requirement.
• Frame budget. Overview N and zoom N depend on the target GPU's VRAM and on measured tokens-per-frame for the chosen model — needs a number from [AI] Vision-Language Model Benchmark & Qwen Alternatives #8. Target hardware is not fixed yet, so [STREAM] Video Ingestion & Dynamic Frame Sampling Service #1 must treat both budgets as runtime configuration, not compile-time constants.
• Prompt ownership. Does apps/stream emit a ready-made pre-prompt string, or only structured evidence that apps/ai/orchestrator templates? Recommendation: structured evidence only — prompt wording belongs with the agent, so it can be tuned without redeploying stream.
• Zoom recursion limit. How many times may the agent zoom before we force a conclusion? Needs a cap for worst-case latency.

---

📋 Action Items

• Benchmark frame differencing vs. dense optical flow: throughput and event coverage recall at equal frame budget
• Benchmark adaptive sampling vs. uniform 1 fps baseline at equal frame budget
• A/B the timestamp overlay for its effect on reported-timestamp error
• Benchmark YOLO on/off: accuracy delta vs. added latency and VRAM — then decide
• Lock the evidence schema handed to apps/ai/orchestrator into packages/event-sdk
• Implement the confirmed modules in packages/optics, consumed by apps/stream ([STREAM] Video Ingestion & Dynamic Frame Sampling Service #1)

---

🔗 Related Resources / Docs

• Implements into: [STREAM] Video Ingestion & Dynamic Frame Sampling Service #1 (stream service)
• Depends on: [DATASET] Golden Dataset Curation & Benchmark Ground Truth #5 (Golden Dataset — every benchmark above needs ground truth), [AI] Vision-Language Model Benchmark & Qwen Alternatives #8 (tokens-per-frame figure)
• Şartname: 3. Senaryo §4 Temel Beklentiler (Minimum Statik Yapı ve Akıllı Pipeline Tasarımı), §7 Değerlendirme Kriterleri
• Internal Docs: documents/features/pipeline.md