Released checkpoint: raw MV staging and MV/RMR ablations

[nvnbagrov]

Hi, @daichi1207, thank you for releasing ReMoRa.

I evaluated the released checkpoint from the exact upstream commit
3a456f7
and found a possible issue in the evaluation data flow.

Possible bug

The GOP evaluation code computes MV magnitude maps and attempts to store them in
pending_mv_maps:

• infer.py, lines 356–373

However, I could not find an initialization or consumer for pending_mv_maps,
so the guarded assignment appears to be a no-op.

Instead, the RMR path reads:

• pending_mv_seqs
• pending_i_indices
• pending_gop_boundaries

Here:

• llava_arch.py, lines 445–460

These values are then passed to the two-stage compressor:

• llava_arch.py, lines 313–323

I do not see these raw-MV fields being populated by the released evaluation
script. Consequently, motion_vectors, gop_boundaries, and
i_frame_indices appear to reach the compressor as None.

I added an opt-in fix that stages these fields without changing the existing
magnitude/GOP logic. I also added two causal ablations:

• Zero MV: RMR receives zero input, producing RMR(0) + 0.
• Zero RMR: the complete RMR output is zeroed before GOP fusion.

Patch and full results:

• Minimal commit
• Evaluation report

Results

All arms used the released checkpoint, complete five-dataset evaluation
(33,915 questions), GOP loading, and the repository's prompts, decoding, and
scoring.

Dataset	Paper	No fix	Real MV + fix	Zero MV	Zero RMR
LongVideoBench	60.8	58.19	58.04	57.89	58.04
MLVU	72.1	69.78	69.60	69.50	69.60
NExT-QA	84.2	82.38	82.39	82.37	82.43
PerceptionTest	67.7	66.91	67.16	67.24	67.32
Video-MME	64.4	62.96	62.67	62.70	62.52
Macro	69.84	68.04	67.97	67.94	67.98

Against the fixed real-MV arm:

Ablation	Macro change	95% video-clustered CI	McNemar p
Zero MV	−0.031 pp	[−0.182, +0.120]	.651
Zero RMR	+0.009 pp	[−0.151, +0.170]	.127

Removing the MV input or the complete RMR output therefore causes no measurable
accuracy decrease. The interventions change only about 1.1–1.2% of predictions.

Could you clarify:

1. How were the raw-MV fields populated during the paper evaluation?
2. Was another evaluation script or configuration used for Table 1?
3. Did you perform a similar causal MV/RMR ablation? Do these results suggest
   that the released model has learned to mostly ignore the MV signal, or is
   there another intended MV path missing from the released evaluation code?
4. Are additional checkpoint files or preprocessing steps required to reproduce
   the reported scores?

Thank you!

[daichi1207]

Thank you for the bug report! I have confirmed the staging issue you identified. I will review the released code and will follow up here.

[nvnbagrov]

Thanks for confirming the staging issue. I prepared a standalone fix that stages the raw MV sequence, I-frame indices, and GOP boundaries during evaluation: #2

The PR contains only the staging fix—none of the ablation code.

I’m also looking forward to hearing the results of your MV check, particularly whether the released checkpoint shows a measurable dependence on the MV/RMR signal.

[daichi1207]

Thanks for the contribution! I’ll merge the staging fix for now. I’ll also check it on my side share any findings.