World-model quality benchmark (JEPA + Reactive vs Combined) - #191
World-model quality benchmark (JEPA + Reactive vs Combined)#191ShauryaVM wants to merge 3 commits into
Conversation
Add quality-side evaluation for the World Model to complement the existing speed benchmark: JEPA reconstruction metrics, null-relative improvement, Reactive vs Combined trajectory impact, and open-loop ADE/FDE pairing helpers. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
|
Under the contribution standard proposed in #198, I consider this PR AI slop in its current form. It presents itself as a quality benchmark, but only evaluates synthetic target-plus-noise tensors and a randomly initialized model; it reports no result from real data or a trained checkpoint. This is not ready for review; please provide the required real-world validation before resubmitting. |
Default quality bench now trains Combined (IL+JEPA) then measures Reactive vs Combined ADE@3s and JEPA recon. Includes a CPU result JSON and a --shard-dir path for packed KITScenes checkpoints. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Pushed a follow-up so this is no longer a random-init / synthetic-only smoke. What changed
Numbers from this machine (CPU, mock backbone, 12 steps, seed 0; JSON in
Combined loss 0.462 → 0.389. JEPA relative improvement vs a zero predictor is 0 on random frames (expected without real video). I am not claiming a KITScenes paper number — this is a trained Combined A/B so the ADE pair is defined. Happy to re-run on a shard you point me at. |
The existing speed benchmark covers FPS/latency for Combined, but we didn’t have a quality side for the world-model path. This adds that.
What’s in here:
Model/evaluation/world_model_quality_benchmark.py(--synthetic/--impact)QUALITY_BENCHMARKS.mdand a link from the speed-benchmark READMEAction-conditioned WM wiring and reasoning-confidence planner coupling are intentionally left for follow-ups.
How I tested
pytest Model/tests/test_world_model_quality.pypython Model/evaluation/world_model_quality_benchmark.py --synthetic