Physical-Intelligence
PI0 model have poor performence on my piper dataset
Based on the excellent pi0 work, we hope to be able to deploy it on our own robot arm. I am using Agilex Piper robot arm data to perform LoRA fine-tuning on the pi0 base model. However, my test results show that the robot arm cannot recognize the target and cannot move the gripper close to the it, and the last joint of the robot arm is always abnormally bent. What could be the cause of this?
I use two images as input: one showing the scene and the other a wrist camera image.
In addition, I use two 24GB RTX 3090 GPUs for training.
@SJYYounge Have you solved this issue - "the robot arm cannot recognize the target and cannot move the gripper close to the it"?
I recently discovered that my previous mistakes were mainly caused by the inconsistency between the test input state and the training state. However, after modifying the input representation, the predicted actions are too small, causing the robotic arm to barely move. Currently, I suspect that using the change in degrees of freedom as the action results in values that are too small in magnitude. Although the model's loss decreased well after LoRA fine-tuning, the model's performance remains poor.
This is part of my stats.json: "action": { "mean": [ -4.333903529484535e-08, 0.0020157869522443835, 0.0013200944144747932, -4.214290422563759e-05, 0.011695201426789108, -0.00013795478592930054 ], "std": [ 1.0091413612500754e-08, 0.0008448569665757645, 0.0003226171148633483, 3.4832450887706154e-05, 0.004836712872686363, 5.273882533245786e-05 ] ... }
Is the action horizon set too small? I encountered a similar situation you described, and increasing the action horizon helped.
What was your batch size set to? Will a relatively small batch size setting have a significant impact on the model's LoRA fine-tuning results? I currently have it set to 8, but the fine-tuning results are not ideal.
Did you deploy it in simulation? What are the degrees of freedom of your own arm? "after modifying the input representation", how do you modify it?
Yes, the simulation was conducted in Gym. After I modified the action and state representations to an end-effector–based representation, this issue no longer occurs. The current main problem is that the model cannot recognize the target. Could the batch size significantly affect the model’s performance? At present, I am fine-tuning using two GPUs with a batch size of 8, but the results are not satisfactory.
I would like to know how you “modified the action and state representations to an end-effector–based representation” ？thx! I have the same problem, finetuned on A100 with bs=32. After the problem is solved, I am willing to try to lower the BS and see how it performs.
Similar to the official testing approach provided for LIBERO, I modified the robot’s state and action representations to be based on the end-effector’s position and orientation. For the rotation representation, I referred to the official LIBERO example file (/examples/libero/main.py) and used the axis–angle representation. This modification effectively resolved the issue where the robot barely moved when using joint degrees of freedom for state and action representation, which was mainly caused by the small magnitude of joint-space actions.
The abnormal bending of the arm mainly resulted from a mismatch between the training and testing representations — my training data used joint-space representations for actions and states, while during testing, I used the end-effector representation as the input to the policy network. As a result, some unrealistic joint motions occurred. Currently, during the initial execution phase, the joints generally follow the directions in the dataset, but in later stages, the arm still struggles to reach the specified task positions.
Thank you for your reply! Hope this helps, and I’m also curious to hear about your results after trying different batch sizes!
Thank you for your reply! I use Aloha's HDF5 as the data collection structure, so it is different from yours. I will continue to check again.
Thank you for your response. And i would like to know how many data samples did you use for fine-tuning? Does each task require a certain amount of data for training? By 'each task', do you mean data from the same scene and with the same language instruction?
