Result gap between official RoboCasa specialist eval and my fine-tuned result from official pretrained checkpoint

[otinus1code]

Hi, thanks for open-sourcing this project.

I am trying to reproduce/fine-tune on RoboCasa and I found a large gap between:

1. directly evaluating the official RoboCasa specialist checkpoint, and
2. evaluating my own fine-tuned checkpoint starting from the official pretrained model.

What I observed

• Direct eval of the official specialist checkpoint Being-H05-2B_robocasa gives about 54% success rate
• The paper / released result is about 56%, so this looks reasonably close
• However, my own fine-tuned checkpoint only gets about 24.67% on the same RoboCasa eval setting

So the eval pipeline itself seems mostly correct, but my fine-tuning result is much worse than the released specialist model.

My setup

• Base pretrained checkpoint used for fine-tuning: Being-H05-2B
• Eval benchmark: RoboCasa
• Number of eval episodes: 1200 total (24 tasks x 50 trials)
• My train and eval log files are attached

Key details from my inspection

• The official specialist checkpoint Being-H05-2B_robocasa can be evaluated normally
• My training script currently initializes from the general checkpoint Being-H05-2B
• During training, action_chunk_length seems to be overridden from 16 to 15
• Eval uses RoboCasa metadata with metadata-variant=merged

Main question

Is the released RoboCasa specialist model trained:

• directly from Being-H05-2B, or
• from another intermediate checkpoint / recipe?

And are there any important training settings needed to reproduce the released RoboCasa specialist performance, such as:

• exact dataset variant
• metadata/statistics file
• action chunk length
• normalization setup
• optimizer / schedule / number of steps
• whether to start from specialist instead of general checkpoint

Additional question

Would you expect a large drop like 54% -> 24.67% if I fine-tune from the general checkpoint with a slightly different recipe, or does this suggest that I may be missing a critical training/eval setting?

Thanks a lot.

train.log

eval.log