-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.html
More file actions
1070 lines (957 loc) · 60.2 KB
/
Copy pathindex.html
File metadata and controls
1070 lines (957 loc) · 60.2 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Gated Memory Policy</title>
<!-- Google Fonts: only external network dependency -->
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@300;400;450;500;600;700;800&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet">
<link rel="icon" type="image/png" href="assets/favicon.png">
<link rel="stylesheet" href="style.css">
</head>
<body>
<!-- STICKY TOC NAV (update href targets and labels to match your sections) -->
<nav id="toc-nav" aria-label="Page sections">
<div class="toc-inner">
<p class="toc-title">On This Page</p>
<a class="toc-link" href="#abstract">Abstract</a>
<a class="toc-link" href="#cross-trial">Cross-Trial Tasks</a>
<a class="toc-link" href="#in-trial">In-Trial Tasks</a>
<a class="toc-link" href="#in-the-wild">Generalization</a>
<a class="toc-link" href="#method">Method</a>
<a class="toc-link" href="#attention">Attention Visualization</a>
<a class="toc-link" href="#benchmark">Benchmark</a>
<a class="toc-link" href="#faq">Questions & Answers</a>
<a class="toc-link" href="#team">Team</a>
<a class="toc-link" href="#acknowledgments">Acknowledgments</a>
</div>
</nav>
<!-- HERO TITLE CARD (paper title, authors, institution, link buttons) -->
<div id="hero-title">
<!-- <p class="hero-venue">RSS 2026</p> -->
<h1 class="paper-title">Gated Memory Policy</h1>
<p class="authors">
<a href="https://yihuai-gao.github.io/" target="_blank" rel="noopener">Yihuai Gao</a>,
Jinyun Liu,
<a href="https://shuangli59.github.io/" target="_blank" rel="noopener">Shuang Li</a>,
<a href="https://shurans.github.io/" target="_blank" rel="noopener">Shuran Song</a><span class="corresponding" title="Corresponding author">†</span>
</p>
<p class="institution">Stanford University</p>
<p class="hero-tagline">Robot policies that blindly extend observation histories overfit and regress. GMP learns <strong>when</strong> and <strong>what</strong> to recall, achieving <span class="highlight">30.1% higher success rates</span> on non-Markovian manipulation while staying competitive on Markovian tasks.</p>
<div class="link-buttons">
<!-- Paper: update href to the newest PDF under paper/ on each iteration -->
<a href="paper/Gated_Memory_Policy_2026-04-22.pdf" target="_blank" rel="noopener" class="btn-link">
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"
stroke-linecap="round" stroke-linejoin="round">
<path d="M14 2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12a2 2 0 0 0 2-2V8z"/>
<polyline points="14 2 14 8 20 8"/>
<line x1="16" y1="13" x2="8" y2="13"/>
<line x1="16" y1="17" x2="8" y2="17"/>
</svg>
Paper
</a>
<!-- arXiv -->
<a href="https://arxiv.org/abs/2604.18933" target="_blank" rel="noopener" class="btn-link">
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"
stroke-linecap="round" stroke-linejoin="round">
<path d="M4 19.5A2.5 2.5 0 0 1 6.5 17H20"/>
<path d="M6.5 2H20v20H6.5A2.5 2.5 0 0 1 4 19.5v-15A2.5 2.5 0 0 1 6.5 2z"/>
<line x1="9" y1="9" x2="15" y2="9"/>
<line x1="9" y1="13" x2="13" y2="13"/>
</svg>
arXiv
</a>
<!-- GitHub: filled / inverted as primary CTA -->
<a href="#" class="btn-link">
<svg viewBox="0 0 24 24" fill="currentColor">
<path d="M12 2C6.477 2 2 6.477 2 12c0 4.42 2.865 8.17 6.839 9.49.5.092.682-.217.682-.482
0-.237-.008-.866-.013-1.7-2.782.603-3.369-1.342-3.369-1.342-.454-1.155-1.11-1.462
-1.11-1.462-.908-.62.069-.608.069-.608 1.003.07 1.531 1.03 1.531 1.03.892 1.529
2.341 1.087 2.91.832.092-.647.35-1.088.636-1.338-2.22-.253-4.555-1.11-4.555-4.943
0-1.091.39-1.984 1.029-2.683-.103-.253-.446-1.27.098-2.647 0 0 .84-.269 2.75 1.025
A9.578 9.578 0 0 1 12 6.836c.85.004 1.705.114 2.504.336 1.909-1.294 2.747-1.025
2.747-1.025.546 1.377.203 2.394.1 2.647.64.699 1.028 1.592 1.028 2.683 0 3.842
-2.339 4.687-4.566 4.933.359.309.678.919.678 1.852 0 1.336-.012 2.415-.012 2.741
0 .267.18.578.688.48C19.138 20.167 22 16.418 22 12c0-5.523-4.477-10-10-10z"/>
</svg>
Code
</a>
<!-- Hugging Face model -->
<a href="https://huggingface.co/yihuai-gao/gated-memory-policy/tree/main"
target="_blank" rel="noopener"
class="btn-link btn-link-hf" aria-label="Hugging Face model checkpoints">
<img class="hf-logo" src="assets/images/hf-logo.svg" alt="">
Model
</a>
<!-- Hugging Face dataset -->
<a href="https://huggingface.co/datasets/yihuai-gao/gated-memory-policy"
target="_blank" rel="noopener"
class="btn-link btn-link-hf" aria-label="Hugging Face dataset">
<img class="hf-logo" src="assets/images/hf-logo.svg" alt="">
Dataset
</a>
</div>
<!-- ABSTRACT (folded into hero so the first viewport carries the full
pitch: title, authors, buttons, and the technical summary). -->
<div id="abstract">
<p>Robotic manipulation tasks exhibit varying memory requirements, ranging from Markovian tasks
that require no memory to non-Markovian tasks that depend on historical information spanning
single or multiple interaction trials. Surprisingly, simply extending observation histories of
a visuomotor policy often leads to a significant performance drop due to distribution shift
and overfitting.</p>
<p>To address these issues, we propose <em>Gated Memory Policy</em> (GMP), a visuomotor
policy that learns both <strong>when</strong> to recall memory and <strong>what</strong> to recall. To learn
when to recall memory, GMP employs a learned memory gate mechanism that selectively activates
history context only when necessary, improving robustness and reactivity. To learn what to
recall efficiently, GMP introduces a lightweight cross-attention module that constructs
effective latent memory representations. To further enhance robustness, GMP injects diffusion
noise into historical actions, mitigating sensitivity to noisy or inaccurate histories during
both training and inference.</p>
<p>On our proposed non-Markovian benchmark <em>MemMimic</em>, GMP achieves a
<strong>30.1% average success rate improvement</strong> over long-history baselines, while
maintaining competitive performance on Markovian tasks in RoboMimic. All code and data will
be publicly available.</p>
</div>
</div><!-- /#hero-title -->
<!-- CROSS TRIAL MEMORY (Pushing, Casting, Flinging)
Underline tab bars, one panel visible at a time.
Video path: assets/videos/cross_trial/[task]/[policy]/[config].mp4
policies: ours, long_hist_dp, no_hist_dp -->
<section id="cross-trial">
<h2>Cross-Trial Memory Tasks</h2>
<p class="body-text">In cross-trial tasks, a physical parameter (such as surface friction or object mass) is randomly sampled per episode and hidden from the policy. The robot is given multiple trials and must leverage outcomes from earlier attempts to adapt. GMP's memory gate selectively activates history context only when it is informative, enabling robust cross-trial adaptation without overfitting to irrelevant past observations.</p>
<!-- 4B CASTING: 2 friction regimes. 3 column video grid, persistent
success rate chart on the right.
Video: assets/videos/cross_trial/casting/[high|low_friction]/
Chart: assets/charts/cross_trial/casting/success_rate.svg -->
<div class="subsection" id="cross-trial-casting">
<div class="task-header">
<div class="task-header-text">
<h3>Iterative Casting (Real)</h3>
<p class="subsection-intro">The robot needs to cast the object with an unknown friction coefficient so that the object stops sliding between two green lines.
The robot stops at the same position in each trial and adaptively adjusts its casting speed. In each episode, the robot is allowed 3 casting attempts. The
episode is considered successful if the last 2 trials are successful.</p>
</div>
<figure class="task-chart">
<div class="col-header">Success Rate</div>
<img src="assets/charts/cross_trial/casting/success_rate.svg" alt="Success rate bar chart — Casting">
</figure>
</div><!-- /.task-header -->
<div class="task-videos-full">
<div class="tab-bar" role="tablist" data-tabgroup="casting">
<button class="tab-btn active" data-tab="high_friction" role="tab">High Friction</button>
<button class="tab-btn" data-tab="low_friction" role="tab">Low Friction</button>
</div>
<div class="tab-panel active" data-tabgroup="casting" data-config="high_friction">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/casting/ours/ours_high_friction.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/casting/long_hist_dp/long_hist_high_friction.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/casting/no_hist_dp/no_hist_high_friction.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<div class="tab-panel" data-tabgroup="casting" data-config="low_friction">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/casting/ours/ours_low_friction.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/casting/long_hist_dp/long_hist_low_friction.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/casting/no_hist_dp/no_hist_low_friction.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
</div><!-- /.task-videos-full -->
</div><!-- /#cross-trial-casting -->
<!-- 4A PUSHING: 7 friction levels, µ notation in labels.
Drop files into assets/videos/cross_trial/pushing/friction_[VALUE]/ -->
<div class="subsection" id="cross-trial-pushing">
<div class="task-header">
<div class="task-header-text">
<h3>Iterative Pushing (Sim)</h3>
<p class="subsection-intro">In each episode, the friction coefficient of the cube is randomly sampled from 0.005 to 0.015
and is unknown during testing. The policy is given 6 trials to push the object to the target region (red box). An episode is considered successful if the cube stops within the target region in all of the last 3 trials.</p>
</div>
<figure class="task-chart">
<div class="col-header">Success Rate</div>
<img src="assets/charts/cross_trial/pushing/success_rate.svg" alt="Success rate bar chart — Pushing">
</figure>
</div><!-- /.task-header -->
<div class="task-videos-full">
<div class="tab-bar" role="tablist" data-tabgroup="pushing">
<button class="tab-btn active" data-tab="friction_0.005" role="tab">μ = 0.005</button>
<button class="tab-btn" data-tab="friction_0.007" role="tab">μ = 0.007</button>
<button class="tab-btn" data-tab="friction_0.009" role="tab">μ = 0.009</button>
<button class="tab-btn" data-tab="friction_0.011" role="tab">μ = 0.011</button>
<button class="tab-btn" data-tab="friction_0.013" role="tab">μ = 0.013</button>
<button class="tab-btn" data-tab="friction_0.015" role="tab">μ = 0.015</button>
</div>
<div class="tab-panel active" data-tabgroup="pushing" data-config="friction_0.005">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/pushing/ours/0.005.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/pushing/long_hist_dp/0.005.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/pushing/no_hist_dp/0.005.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<div class="tab-panel" data-tabgroup="pushing" data-config="friction_0.007">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/pushing/ours/0.007.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/pushing/long_hist_dp/0.007.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/pushing/no_hist_dp/0.007.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<div class="tab-panel" data-tabgroup="pushing" data-config="friction_0.009">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/pushing/ours/0.009.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/pushing/long_hist_dp/0.009.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/pushing/no_hist_dp/0.009.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<div class="tab-panel" data-tabgroup="pushing" data-config="friction_0.011">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/pushing/ours/0.011.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/pushing/long_hist_dp/0.011.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/pushing/no_hist_dp/0.011.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<div class="tab-panel" data-tabgroup="pushing" data-config="friction_0.013">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/pushing/ours/0.013.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/pushing/long_hist_dp/0.013.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/pushing/no_hist_dp/0.013.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<div class="tab-panel" data-tabgroup="pushing" data-config="friction_0.015">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/pushing/ours/0.015.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/pushing/long_hist_dp/0.015.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/pushing/no_hist_dp/0.015.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
</div><!-- /.task-videos-full -->
</div><!-- /#cross-trial-pushing -->
<!-- 4C FLINGING: cloth mass levels.
Drop files into assets/videos/cross_trial/flinging/[mass_level]/ -->
<div class="subsection" id="cross-trial-flinging">
<div class="task-header">
<div class="task-header-text">
<h3>Iterative Flinging (Sim)</h3>
<p class="subsection-intro">The robot must fling the cloth so that its far edge lands in the target (black) area.
In each episode, the cloth's mass is randomly sampled between 0.1 kg and 2.0 kg, and the robot is allowed five flings. Flinging too slowly prevents the cloth from fully extending, while flinging too fast causes it to fold back on itself—both resulting in failure. An episode is considered successful if the last 3 trials are all successful.</p>
</div>
<figure class="task-chart">
<div class="col-header">Success Rate</div>
<img src="assets/charts/cross_trial/flinging/success_rate.svg" alt="Success rate bar chart — Flinging">
</figure>
</div><!-- /.task-header -->
<div class="task-videos-full">
<div class="tab-bar" role="tablist" data-tabgroup="flinging">
<button class="tab-btn active" data-tab="mass_0.1" role="tab">0.1 kg</button>
<button class="tab-btn" data-tab="mass_0.15" role="tab">0.15 kg</button>
<button class="tab-btn" data-tab="mass_0.2" role="tab">0.2 kg</button>
<button class="tab-btn" data-tab="mass_0.3" role="tab">0.3 kg</button>
<button class="tab-btn" data-tab="mass_0.5" role="tab">0.5 kg</button>
<button class="tab-btn" data-tab="mass_1.0" role="tab">1.0 kg</button>
<button class="tab-btn" data-tab="mass_2.0" role="tab">2.0 kg</button>
</div>
<div class="tab-panel active" data-tabgroup="flinging" data-config="mass_0.1">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/flinging/ours/0.1.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/flinging/long_hist_dp/0.1.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/flinging/no_hist_dp/0.1.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<div class="tab-panel" data-tabgroup="flinging" data-config="mass_0.15">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/flinging/ours/0.15.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/flinging/long_hist_dp/0.15.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/flinging/no_hist_dp/0.15.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<div class="tab-panel" data-tabgroup="flinging" data-config="mass_0.2">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/flinging/ours/0.2.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/flinging/long_hist_dp/0.2.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/flinging/no_hist_dp/0.2.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<div class="tab-panel" data-tabgroup="flinging" data-config="mass_0.3">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/flinging/ours/0.3.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/flinging/long_hist_dp/0.3.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/flinging/no_hist_dp/0.3.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<div class="tab-panel" data-tabgroup="flinging" data-config="mass_0.5">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/flinging/ours/0.5.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/flinging/long_hist_dp/0.5.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/flinging/no_hist_dp/0.5.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<div class="tab-panel" data-tabgroup="flinging" data-config="mass_1.0">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/flinging/ours/1.0.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/flinging/long_hist_dp/1.0.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/flinging/no_hist_dp/1.0.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<div class="tab-panel" data-tabgroup="flinging" data-config="mass_2.0">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/flinging/ours/2.0.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/flinging/long_hist_dp/2.0.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/cross_trial/flinging/no_hist_dp/2.0.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
</div><!-- /.task-videos-full -->
</div><!-- /#cross-trial-flinging -->
</section><!-- /#cross-trial -->
<!-- IN TRIAL MEMORY (Cup, Match Color, Place Back)
Video path: assets/videos/in_trial/[task]/[location]/[policy].mp4
policies: ours, long_hist_dp, no_hist_dp -->
<section id="in-trial">
<h2>In-Trial Memory Tasks</h2>
<p class="body-text">In-trial tasks require the robot to recall a cue observed earlier within a single episode (such as an object's starting position or a bin's color). The relevant context appears only once; the policy must encode and retain it to succeed at a later decision point. GMP's lightweight cross-attention module constructs effective latent memory on demand, gating out irrelevant history to stay focused.</p>
<!-- 5A CUP: multiple locations, initial position accordion below.
Video: assets/videos/in_trial/place_back_real/[loc_N]/
Overlay: assets/images/cup_overlay/[loc_N]/initial_pos.png
In the wild: assets/videos/in_trial/place_back_real/in_the_wild_ours.mp4 -->
<div class="subsection" id="in-trial-place-back-real">
<div class="task-header">
<div class="task-header-text">
<h3>Continuous Place Back (Real)</h3>
<p class="subsection-intro">A cup and a saucer are randomly placed on the table. The robot must first place the cup on the saucer and then return it to within 5 cm of its original position. This task tests the policy's <strong>spatial memory</strong> and <strong>robustness to real-world noise</strong>.</p>
</div>
<figure class="task-chart">
<div class="col-header">Success Rate</div>
<img src="assets/charts/in_trial/place_back_real/success_rate.svg" alt="Success rate bar chart — Place Back (Real)">
</figure>
</div><!-- /.task-header -->
<div class="task-videos-full">
<div class="tab-bar" role="tablist" data-tabgroup="place-back-real">
<button class="tab-btn active" data-tab="loc_1" role="tab">Location 1</button>
<button class="tab-btn" data-tab="loc_2" role="tab">Location 2</button>
<button class="tab-btn" data-tab="loc_3" role="tab">Location 3</button>
<button class="tab-btn" data-tab="loc_4" role="tab">Location 4</button>
<button class="tab-btn" data-tab="loc_5" role="tab">Location 5</button>
</div>
<div class="tab-panel active" data-tabgroup="place-back-real" data-config="loc_1">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back_real/loc_1/ours.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back_real/loc_1/long_hist_dp.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back_real/loc_1/no_hist_dp.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<div class="tab-panel" data-tabgroup="place-back-real" data-config="loc_2">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back_real/loc_2/ours.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back_real/loc_2/long_hist_dp.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back_real/loc_2/no_hist_dp.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<div class="tab-panel" data-tabgroup="place-back-real" data-config="loc_3">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back_real/loc_3/ours.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back_real/loc_3/long_hist_dp.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back_real/loc_3/no_hist_dp.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<div class="tab-panel" data-tabgroup="place-back-real" data-config="loc_4">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back_real/loc_4/ours.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back_real/loc_4/long_hist_dp.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back_real/loc_4/no_hist_dp.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<div class="tab-panel" data-tabgroup="place-back-real" data-config="loc_5">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back_real/loc_5/ours.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back_real/loc_5/long_hist_dp.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back_real/loc_5/no_hist_dp.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<details class="overlay-details">
<summary>Show Initial Cup Position Distribution</summary>
<figure>
<img src="assets/images/pos_overlay.png" alt="Initial cup position distribution across all locations">
<figcaption>Initial cup position distribution across all locations</figcaption>
</figure>
</details>
</div><!-- /.task-videos-full -->
</div><!-- /#in-trial-place-back-real -->
<!-- 5B MATCH COLOR: 4 locations x 3 policies.
Drop files: assets/videos/in_trial/match_color/[policy]/[0-3].mp4 -->
<div class="subsection" id="in-trial-match-color">
<div class="task-header">
<div class="task-header-text">
<h3>Match Color (Sim)</h3>
<p class="subsection-intro">The robot picks up a cube from one of 4 bins with randomly assigned colors. Once the cube is picked up, the bin colors are shuffled, and the robot needs to put the cube back in the bin with the original color. This task tests the policy's <strong>visual memory</strong>.</p>
</div>
<figure class="task-chart">
<div class="col-header">Success Rate</div>
<img src="assets/charts/in_trial/match_color/success_rate.svg" alt="Success rate bar chart — Match Color">
</figure>
</div><!-- /.task-header -->
<!-- Location tabs -->
<div class="task-videos-full">
<div class="tab-bar" role="tablist" data-tabgroup="match-color">
<button class="tab-btn active" data-tab="loc_1" role="tab">Config 1</button>
<button class="tab-btn" data-tab="loc_2" role="tab">Config 2</button>
<button class="tab-btn" data-tab="loc_3" role="tab">Config 3</button>
<button class="tab-btn" data-tab="loc_4" role="tab">Config 4</button>
</div>
<div class="tab-panel active" data-tabgroup="match-color" data-config="loc_1">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/match_color/ours/0.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/match_color/long_hist_dp/0.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/match_color/no_hist_dp/0.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<div class="tab-panel" data-tabgroup="match-color" data-config="loc_2">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/match_color/ours/1.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/match_color/long_hist_dp/1.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/match_color/no_hist_dp/1.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<div class="tab-panel" data-tabgroup="match-color" data-config="loc_3">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/match_color/ours/2.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/match_color/long_hist_dp/2.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/match_color/no_hist_dp/2.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<div class="tab-panel" data-tabgroup="match-color" data-config="loc_4">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/match_color/ours/3.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/match_color/long_hist_dp/3.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/match_color/no_hist_dp/3.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
</div><!-- /.task-videos-full -->
</div><!-- /#in-trial-match-color -->
<!-- 5C PLACE BACK: 4 locations x 3 policies.
Drop files: assets/videos/in_trial/place_back/[policy]/[0-3].mp4 -->
<div class="subsection" id="in-trial-place-back">
<div class="task-header">
<div class="task-header-text">
<h3>Discrete Place Back (Sim)</h3>
<p class="subsection-intro">A cube is randomly placed in one of four bins, and the robot picks it up, holds it in the air for 2 seconds before returning it to the original bin. This task tests the policy's <strong>spatial memory</strong>.</p>
</div>
<figure class="task-chart">
<div class="col-header">Success Rate</div>
<img src="assets/charts/in_trial/place_back/success_rate.svg" alt="Success rate bar chart — Place Back">
</figure>
</div><!-- /.task-header -->
<div class="task-videos-full">
<div class="tab-bar" role="tablist" data-tabgroup="place-back">
<button class="tab-btn active" data-tab="loc_1" role="tab">Config 1</button>
<button class="tab-btn" data-tab="loc_2" role="tab">Config 2</button>
<button class="tab-btn" data-tab="loc_3" role="tab">Config 3</button>
<button class="tab-btn" data-tab="loc_4" role="tab">Config 4</button>
</div>
<div class="tab-panel active" data-tabgroup="place-back" data-config="loc_1">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back/ours/0.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back/long_hist_dp/0.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back/no_hist_dp/0.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<div class="tab-panel" data-tabgroup="place-back" data-config="loc_2">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back/ours/1.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back/long_hist_dp/1.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back/no_hist_dp/1.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<div class="tab-panel" data-tabgroup="place-back" data-config="loc_3">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back/ours/2.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back/long_hist_dp/2.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back/no_hist_dp/2.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
<div class="tab-panel" data-tabgroup="place-back" data-config="loc_4">
<div class="video-grid three-col">
<div class="video-col">
<div class="col-header ours">Ours (Gated Memory)</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back/ours/3.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">Long-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back/long_hist_dp/3.mp4" type="video/mp4"></video></figure>
</div>
<div class="video-col">
<div class="col-header">No-History DP</div>
<figure><video autoplay loop muted playsinline preload="none"><source src="assets/videos/in_trial/place_back/no_hist_dp/3.mp4" type="video/mp4"></video></figure>
</div>
</div>
</div>
</div><!-- /.task-videos-full -->
</div><!-- /#in-trial-place-back -->
</section><!-- /#in-trial -->
<!-- IN THE WILD: own chapter because the grid runs full-bleed across the
viewport. 40 outdoor trials in an 8x5 tile generated by
scripts/make_in_the_wild_grid.py. -->
<section id="in-the-wild">
<span class="section-label">Generalization</span>
<h2>In-the-Wild Results</h2>
<p class="body-text">40 outdoor Continuous Place Back trials run end-to-end with Gated Memory Policy, across backgrounds, lighting, surfaces, and cup placements never seen in training. Some include <span class="highlight">human perturbation</span>, where we flip the cup over by hand mid-task.</p>
<!-- Left half of the grid (cols 0-3) = no-perturbation trials;
right half (cols 4-7) = perturbation trials. Labels mirror that. -->
<div class="wild-labels">
<div class="wild-label">
<span class="wild-label-eyebrow">Continuous Place Back</span>
<span class="wild-label-title">Without Perturbation</span>
</div>
<div class="wild-label">
<span class="wild-label-eyebrow">Continuous Place Back</span>
<span class="wild-label-title">With Perturbation</span>
</div>
</div>
<!-- preload="auto": this is the banner for its own chapter and ~17MB;
warming the buffer from page load means the native <video loop>
restart from frame 0 has no re-buffer gap. -->
<video class="full-bleed" autoplay loop muted playsinline preload="auto">
<source src="assets/videos/in_trial/place_back_real/in_the_wild_ours.mp4" type="video/mp4">
</video>
</section>
<!-- METHOD (Architecture + Gating Mechanism, zigzag layout) -->
<section id="method">
<h2>Method</h2>
<div class="method-row">
<figure class="method-img">
<img src="assets/images/architecture.svg" alt="Model architecture diagram">
</figure>
<div class="method-caption">
<h3>Model Architecture</h3>
<p>The gated attention module features three key designs: (1) Binary memory gate μ<sub>t</sub> that determines whether history cross-attention is skipped or applied. (2) Noised history action condition to improve robustness and reduce overfitting. (3) Cached history tokens during inference time to reduce computational cost.</p>
</div>
</div>
<div class="method-row method-row-flip">
<figure class="method-img">
<img src="assets/images/gating_training.svg" alt="Gating training pipeline">
</figure>
<div class="method-caption">
<h3>Gating Mechanism</h3>
<p>
We split the dataset into <em>D</em><sub>train</sub> and <em>D</em><sub>val</sub>.
<strong>1)</strong> Train two policies on <em>D</em><sub>train</sub> with memory gate always off
<span class="math-gray">π</span> or always on <span class="math-blue">π<sub>mem</sub></span>.
<strong>2)</strong> Evaluate the two policies on <em>D</em><sub>val</sub> for <em>N</em> rounds and
calculate the error of the predicted actions at each timestep <em>t</em>.
<strong>3)</strong> At timestep <em>t</em>, if the error of the no-memory policy
<span class="math-gray">δ<sub>t</sub></span> is significantly larger than the error of the
memory policy <span class="math-blue">δ<sup>mem</sup><sub>t</sub></span>, we label
<em>memory-required</em> (μ<sub>t</sub>=1) for this timestep; otherwise, μ<sub>t</sub>=0.
This stage is referred to as <em>Calibration</em> of the memory gate.
<strong>4)</strong> We freeze the memory gate weights and retrain the policy on the full
dataset to obtain the final Gated Memory Policy π<sub>gated</sub>.
</p>
</div>
</div>
</section>
<!-- INTERPRETABILITY: attention visualizations
Combined cube pushing (cross trial) and match color (in trial). -->
<section id="attention">
<h2>What Is the Policy Attending To?</h2>
<p class="body-text">We visualize causal cross-attention weights to understand which past timesteps the policy attends to when computing the next action. The gated memory mechanism actively selects the most informative historical context, confirming that the policy learns structured, interpretable recall rather than attending uniformly across history.</p>
<div class="tab-bar" role="tablist" data-tabgroup="attn-vis">
<button class="tab-btn active" data-tab="cube-pushing" role="tab">Cube Pushing</button>
<button class="tab-btn" data-tab="match-color" role="tab">Match Color</button>
</div>
<div class="tab-panel active" data-tabgroup="attn-vis" data-config="cube-pushing">
<figure>
<video autoplay loop muted playsinline controls preload="none">
<source src="assets/videos/attention_vis_cube_pushing.mp4" type="video/mp4">
</video>
<figcaption>Visualization of gated cross-attention weights for each 8-step action trajectory throughout one episode. <span class="math-blue">Blue</span> indicates the memory gate is on (μ<sub>t</sub>=1) and <span class="math-gray">gray</span> indicates the memory gate is off (μ<sub>t</sub>=0). During the pushing stage in Trial 4, attention focuses primarily on the outcomes from Trial 3 (undershoot) and Trial 2 (overshoot). The policy uses these past outcomes to adjust its action and achieve a successful push.</figcaption>
</figure>
</div>
<div class="tab-panel" data-tabgroup="attn-vis" data-config="match-color">
<figure>
<video autoplay loop muted playsinline controls preload="none">
<source src="assets/videos/attention_vis_match_color.mp4" type="video/mp4">
</video>
<figcaption>We visualize the gated cross-attention weights for each 8-step action chunk. At <em>t</em>=80, when the robot is placing the cube back, the attention focuses primarily on <em>t</em>=48, when the robot first observed the bin colors. <span class="math-blue">Blue</span> indicates the memory gate is on (μ<sub>t</sub>=1); <span class="math-gray">gray</span> indicates the gate is off (μ<sub>t</sub>=0).</figcaption>
</figure>
</div>
</section>
<!-- BENCHMARK PERFORMANCE: two stacked blocks (RoboMimic, Mikasa Robo) -->
<section id="benchmark">
<h2>Benchmark Performance</h2>
<div class="benchmark-block">
<h3>RoboMimic Results</h3>
<figure class="benchmark-chart">
<img src="assets/charts/robomimic_bar_chart_new.svg" alt="RoboMimic benchmark success rates across tasks">
</figure>
<p class="benchmark-caption">We evaluate 3 tasks from the RoboMimic benchmarks: <em>Tool Hang, Square, and Transport</em>. (ph, mh) indicate that the data were collected from proficient-human (ph) or multi-human (mh). While most long-history policies experience performance drops on these Markovian tasks, GMP maintains competitive performance by leveraging the gating mechanism.</p>
</div>
<div class="benchmark-block">
<h3>Mikasa-Robo Results</h3>
<figure class="benchmark-chart">
<img src="assets/charts/mikasa_benchmark.svg" alt="Mikasa-Robo benchmark success rates across 5 memory-dependent tasks">
</figure>
<p class="benchmark-caption">We evaluate 5 tasks on the MIKASA-Robo benchmark, outperforming prior work MemoryVLA by 26.6% on average. The baseline performance statistics are reported in MIKASA-Robo and MemoryVLA.</p>
</div>
</section>
<!-- FAQ
Edit each question and answer. Add or remove .faq-item blocks.
Pure <details>/<summary>, no JS. -->
<section id="faq">
<h2>Questions & Answers</h2>
<div class="faq-list">
<details class="faq-item">
<summary>When is the memory gate helpful, and when should I train it?</summary>
<div class="faq-answer">
<p>The memory gate helps when the policy significantly overfits to history observations. This typically happens in tasks that require high precision manipulation and where there is not enough data to cover the expanded input space that memory introduces. A concrete example is <em>Tool Hang</em> in RoboMimic. The gate also helps when reactiveness matters: it reduces runtime and makes the model more efficient. That said, for most of the memory intensive tasks in our benchmark, adding the gate does not significantly improve performance. A practical approach is to first train the model without a gate (a simplified version of GMP), run evaluation, and then add gates when the model fails due to the issues above.</p>
</div>
</details>
<details class="faq-item">
<summary>What is the inference overhead of GMP vs. a no-memory policy?</summary>
<div class="faq-answer">
<p>We only include a single aggregated token from a ViT per image (e.g., the class token in CLIP or the MAP token in SigLIP), and all history tokens are cached. The cross attention module also scales linearly in history length. As a result, the total inference time is similar to that of a policy without memory. We release an in the wild checkpoint so anyone can try it in the real world.</p>
</div>
</details>
<details class="faq-item">
<summary>What should I do if I have trouble reproducing the results?</summary>
<div class="faq-answer">
<p>Please read through the <code>README.md</code> in each folder of the codebase carefully. The experiment details in the paper's appendix will also be helpful. If your problem is still not solved, feel free to submit a GitHub issue with a detailed description.</p>
</div>
</details>
<details class="faq-item">
<summary>How long can our model memorize?</summary>
<div class="faq-answer">
<p>Due to the strong scalability of cross attention, the history length is only bottlenecked by training GPU memory. By caching visual features and freezing a pretrained image encoder, our model is able to attend to 6,000 image frames (with paired actions), achieving memory recall of up to 10 minutes at 10 fps. Please refer to <strong>Finding 7</strong> for more details.</p>
<div class="faq-video-pair">
<figure class="faq-video">
<video autoplay controls muted playsinline loop preload="metadata">
<source src="assets/videos/faq/long_memory_ep7.mp4" type="video/mp4">
</video>
<figcaption>Episode 7 · ~8 min</figcaption>
</figure>
<figure class="faq-video">
<video autoplay controls muted playsinline loop preload="metadata">
<source src="assets/videos/faq/long_memory_ep69.mp4" type="video/mp4">
</video>
<figcaption>Episode 69 · ~10 min</figcaption>
</figure>
</div>
<p class="faq-video-note">The robot may appear frozen for long stretches, which is expected: we are stress testing the policy's memory duration. At the end, once the bin colors shuffle, the robot places the cube back in its original bin. Both clips play at real time speed (10 fps); drag the progress bar to skim through the full episode.</p>
</div>
</details>
</div>
</section>
<!-- TEAM
Replace each .author-photo svg with
<img src="assets/images/team/firstname_lastname.jpg" alt="Name"> -->
<section id="team">
<h2>Team</h2>
<div class="team-grid">
<a class="author-card" href="https://yihuai-gao.github.io/" target="_blank" rel="noopener">
<div class="author-photo">
<img src="assets/images/team/yihuai_gao.jpg" alt="Yihuai Gao">
</div>
<p class="author-name">Yihuai Gao</p>
<p class="author-affil">Stanford University</p>
</a>
<div class="author-card">
<div class="author-photo">