Repository navigation
A question about finetune process #3
Description
Thanks for your great work!
The paper mentions that the model is fine-tuned using 180 episodes for Cover Blocks and 308 episodes for Put Back Block. I’m wondering what the starting point of this fine-tuning is.
Is the Qwen3.5-4B + Action Expert model trained directly on these 180/308 episodes, with the Action Expert initialized from scratch? Or has the Qwen3.5-4B + Action Expert model already been pretrained on a large-scale robotics dataset before being fine-tuned on these task-specific demonstrations?
Activity
wadeKeith commentedon Sep 29, 2026
Thank you for your interest in SimpleMemVLA!
The 180 episodes for Cover Blocks and 308 episodes for Put Back Block are the task-specific demonstrations used to train our real-world policies. For both tasks, we initialize the vision-language backbone from Qwen3.5-4B, while the DiT-based Action Expert is initialized from scratch.
The two policies are trained independently: one is trained on the 180 Cover Blocks episodes and then deployed. The other starts again from the original Qwen3.5-4B weights with a newly initialized Action Expert, and is trained on the 308 Put Back Block episodes before deployment. Neither policy undergoes additional robotics pretraining beforehand, and no weights are transferred between the two tasks.
SimpleMemVLA pretraining is already underway—stay tuned for updates!
yzysmile commentedon Sep 30, 2026
Thanks for the clarification!
I’m a bit surprised that the real-world demo performance can be so strong when training starts directly from the original Qwen3.5-4B weights with a newly initialized Action Expert, using only 180 episodes for Cover Blocks and 308 episodes for Put Back Block.
Are these numbers sufficient to achieve the demo performance shown in the paper/videos, without any additional robotics pretraining? Also, are there any important training tricks, hyperparameter settings that are critical for achieving this level of performance?
wadeKeith commentedon Sep 30, 2026
Thank you for the follow-up!
Yes, 180 episodes for Cover Blocks and 308 episodes for Put Back Block are the full training datasets used for the real-world policies shown in our paper and videos, with no additional robotics pretraining. To clarify the status of our pretraining work: it is still in preparation, and we have not started it yet.
Personally, I was also a little surprised when we first saw the results! We think they support the usefulness of memory as context for VLA models. As discussed in the paper, even in failure cases, the model can still identify the correct sequence of the red, green, and blue blocks. The failures often involve execution errors, such as unsuccessful grasps. My interpretation is that these reflect limitations in action execution, which could improve with more demonstration data. Correctly recovering the sequence despite these execution failures supports our view that historical video is serving as useful memory context.
Regarding training settings, in our experience, SimpleMemVLA has not been particularly sensitive to hyperparameters, and we trained these policies largely using the default settings in our training setup. That said, this is an empirical observation rather than a conclusion from an extensive hyperparameter study. We encourage you to experiment with different settings, and contributions sharing useful configurations or findings in the repo would be very welcome!
yzysmile commentedon Sep 30, 2026
Thanks for your reply! Looking forward to seeing more great work from you in the future.