Clarification on LAPA-DINOv2 Composite Robot classification reproducibility

[huiyijian]

Hi LARYBench team,

Thanks for releasing the benchmark, code, and public data. We are trying to reproduce the LAPA-DINOv2 classification baseline on
robot_1st / Composite Robot.

According to the leaderboard, LAPA-DINOv2 reports:

Composite Robot: 34.19

We followed the public pipeline and extracted latent actions for robot_1st with dinov2, then trained the classification probe. The
equivalent classify command is:

python -m lary.cli classify \
    --model dinov2 \
    --dataset robot_1st \
    --dim 1024 \
    --classes 54 \
    --batch-size 64 \
    --gpus 0,1,2,3,4,5,6,7

We used a local wrapper only to set paths, WANDB_MODE=offline, NUM_WORKERS=0, and GPU IDs. The wrapper eventually calls the public python -m
lary.cli classify command above. We also made local path-resolution / num_workers patches to make the public data run in our environment;
the classification config and probe training logic were otherwise unchanged.

Our local data statistics are:

train_la_robot_1st_dinov2.csv: 376,895 rows
val_la_robot_1st_dinov2.csv:   161,527 rows
num_classes: 54

train/val video_path overlap: 0
train/val video_path + sample_indices overlap: 0

One unreadable video sample was removed before merging the train LA CSV:

AgiBotWorld-Beta/160335_0.mp4

This affects only 1 / 538,423 samples, so it should not explain a large accuracy difference.

The training log from logs/classification/robot_1st/dinov2/log_r0.csv is:

epoch,train_acc,val_acc
1,68.12081,33.72458
2,86.50353,38.83867
3,90.38617,42.38715
4,92.25835,45.45387
5,93.28162,49.22185
6,94.05411,50.88542
7,94.63660,50.39807
8,95.14266,50.70002
9,95.59459,52.83358
10,95.95735,53.87091
11,96.32064,54.79725

The epoch-1 validation accuracy is very close to the leaderboard number 34.19, but continuing training with the released config increases
validation accuracy to 54.80 by epoch 11. The run stopped at epoch 11 due to an external interruption, but latest.pt was saved successfully
at epoch 11.

Could you please clarify the exact evaluation protocol used for the leaderboard?

Specifically:

1. Does the 34.19 Composite Robot result correspond to epoch 1, final epoch, or best validation epoch?
2. Is classification/configs/eval/vitl/manipulation.yaml the exact config used to produce the leaderboard number?
We can provide the generated log_r0.csv, classification_stats.json, and confusion_matrix.png if helpful.

Thank you very much. I hope it's just my mistake.

Zhaocheng 

[lapa_dinov2_robot_1st_issue_artifacts.tar.gz](https://github.com/user-attachments/files/27468575/lapa_dinov2_robot_1st_issue_artifacts.tar.gz)

[ha8siouhdi]

Thanks for the detailed report and for providing the artifacts! We really appreciate your rigorous reproduction efforts.

To answer your questions:

The 34.19% result for Composite Robot corresponds to the final epoch.

Yes, classification/configs/eval/vitl/manipulation.yaml is the exact config we used to produce the leaderboard number.

Given the significant gap in later epochs, it is highly likely that there are some subtle differences in detailed hyperparameters, environmental setups, or data processing logic.

Would you be open to sharing a contact method (like email or WeChat) so we can align our setups and figure this out together? It would be much faster and easier for us to communicate directly.

I am going to re-run this experiment on our end in the meantime. Thanks again for bringing this up!

[although-not-but]

I want to konw how to clarification Atomic Robot classification

[ha8siouhdi]

@although-not-but By tracking the known motion trajectory, we continuously accumulate the robot's movements. Once this accumulated value exceeds a predefined threshold for a specific action (such as leftward displacement surpassing a set distance), we capture the frame and assign the corresponding label.

[although-not-but]

@ha8siouhdi If I want to make a fair comparison with the Atomic Robot classification results in the paper, do I only need to additionally download the LIBERO dataset and use the CSV files you provide?