conda create -n realrobot python=3.11 -y
pip install -r requirements.txt

# please install with your PyTorch and CUDA version
# e.g. torch 2.4.0 + cuda 118:
pip install torch-geometric  torch-scatter torch-sparse torch-cluster -f https://data.pyg.org/whl/torch-2.4.0+cu118

Note: spconv must matches your CUDA version, see official Github for more information.

# e.g. for CUDA 11.8:
pip3 install spconv-cu118

# build FPS sampling operations (CUDA required)
cd libs/pointops
# docker & multi GPU arch
TORCH_CUDA_ARCH_LIST="ARCH LIST" python setup.py install
# e.g. 7.5: RTX 3000; 8.0: a100 More available in: https://developer.nvidia.com/cuda-gpus
TORCH_CUDA_ARCH_LIST="7.5 8.0" python setup.py install
cd ../..

1. RealSense

   • Install librealsense

     Repeat here:

     sudo apt-get update && sudo apt-get upgrade && sudo apt-get dist-upgrade
     sudo apt-get install libssl-dev libusb-1.0-0-dev libudev-dev pkg-config libgtk-3-dev
     sudo apt-get install git wget cmake build-essential
     sudo apt-get install libglfw3-dev libgl1-mesa-dev libglu1-mesa-dev at
     
     cd /some/path
     git clone https://github.com/IntelRealSense/librealsense.git
     cd IntelRealSense
     ./scripts/setup_udev_rules.sh
     ./scripts/patch-realsense-ubuntu-lts-hwe.sh
     
     mkdir build && cd build
     
     cmake ../ -DCMAKE_BUILD_TYPE=Release
     sudo make uninstall && make clean && make && sudo make install
     

     After installing, you can connect cameras with USB and you can run realsense-viewer to get the serial numbers as well as to visualize results.

   • Install pyrealsense2

     pip install pyrealsense2
     

   • Open realsense-viewer and get the serial numbers of your cameras. Modify the variables here. We use two cameras by default, but it is very easy to adapt to other number of cameras.

1. Low-Cost-Robot

   Please follow the original repo.

To caliberate your camera, you can first print out the charuco file under asserts/calib.io_charuco_297x210_5x7_35_26_DICT_4X4.pdf and put it on your table. Then you can run with:

python scripts/caliberate_cameras.py

The caliberation file will be saved under data/calib.

• For single arm setting, you can run with:

python scripts/record_low_cost_robot_teleop.py task_name=${task_name} user_name=${use_name} episode_id=${episode_id} teleop=low_cost_robot_single_arm teleop.leader.device_name=${device_name} teleop.follower.device_name=${device_name}

• Similarly, for dual arm setting, you can run with:

python scripts/record_low_cost_robot_teleop.py task_name=${task_name} user_name=${use_name} episode_id=${episode_id} teleop=low_cost_robot teleop.op_left.leader.device_name=${device_name} teleop.op_left.follower.device_name=${device_name} teleop.op_right.leader.device_name=${device_name} teleop.op_right.follower.device_name=${device_name}

After each episode, you can press q to end it.

Then the reajectories will be saved under data/teleop/${task_name}/${use_name}/ep_${episode_id} with format of:

├── teleop
│   ├── ...
│   ├── task_name
│   │   ├── use_name
│   │   │   ├── ep_id
|   |   |   │   ├── meta.json   # meta data
|   |   |   │   ├── ${timestamp1}.npy    # recorded data
|   |   |   │   ├── ${timestamp2}.npy
|   |   |   │   ├── ...
│   └── ...

ACT training with RGB images, by default we use 4 gpu with DDP

python src/train.py task_name=${task_name} trainer.devices=4 exp_low_cost_robot=base_rgb task_name=reach_cube

RGB-D reach

python src/train.py task_name=reach_cube exp_low_cost_robot=base_rgbd

Pointcloud reach:

python src/train.py task_name=reach_cube exp_low_cost_robot=base_pcd data.train.calib_file=data/calib/reach.npy

Pick, train for more epochs:

# we use loop to control epoch length, as too much validation may be slow
# you can also use trainer.max_epochs to control epoch number
python src/train.py exp_low_cost_robot=base_rgb data.train.loop=500 exp_name=base_rgb_loop500 task_name=pick_cube

Evaluate reach cube with RGB:

python src/eval_low_cost_robot_act.py exp_low_cost_robot=base_rgb task_name=reach_cube max_timesteps=150 num_rollouts=20 ckpt_path=${ckpt_path} num_rollouts=${num_rollouts}

Evaluate reach cube with point cloud:

python src/eval_low_cost_robot_act.py task_name=reach_cube max_timesteps=150 num_rollouts=20 exp_low_cost_robot=base_pcd data.train.calib_file=data/calib/reach.npy ckpt_path=${ckpt_path} num_rollouts=${num_rollouts}

The script will evaluate the given checkpoint with ${num_rollouts} repeats. You can early stop one episode by press q.

This codebase is based on Hydra, which allows for convenient configuration overriding:

python src/train.py trainer.max_epochs=20 seed=300

Note: You can also add new parameters with + sign.

python src/train.py +some_new_param=some_new_value

# train on CPU
python src/train.py trainer=cpu

# train on 1 GPU
python src/train.py trainer=gpu

# train on TPU
python src/train.py +trainer.tpu_cores=8

# train with DDP (Distributed Data Parallel) (4 GPUs)
python src/train.py trainer=ddp trainer.devices=4

# train with DDP (Distributed Data Parallel) (8 GPUs, 2 nodes)
python src/train.py trainer=ddp trainer.devices=4 trainer.num_nodes=2

# simulate DDP on CPU processes
python src/train.py trainer=ddp_sim trainer.devices=2

# accelerate training on mac
python src/train.py trainer=mps

# train with pytorch native automatic mixed precision (AMP)
python src/train.py trainer=gpu +trainer.precision=16

# gradient clipping may be enabled to avoid exploding gradients
python src/train.py trainer.gradient_clip_val=0.5

# run validation loop 4 times during a training epoch
python src/train.py +trainer.val_check_interval=0.25

# accumulate gradients
python src/train.py trainer.accumulate_grad_batches=10

# terminate training after 12 hours
python src/train.py +trainer.max_time="00:12:00:00"

Note: PyTorch Lightning provides about 40+ useful trainer flags.

# runs 1 epoch in default debugging mode
# changes logging directory to `logs/debugs/...`
# sets level of all command line loggers to 'DEBUG'
# enforces debug-friendly configuration
python src/train.py debug=default

# run 1 train, val and test loop, using only 1 batch
python src/train.py debug=fdr

# print execution time profiling
python src/train.py debug=profiler

# try overfitting to 1 batch
python src/train.py debug=overfit

# raise exception if there are any numerical anomalies in tensors, like NaN or +/-inf
python src/train.py +trainer.detect_anomaly=true

# use only 20% of the data
python src/train.py +trainer.limit_train_batches=0.2 \
+trainer.limit_val_batches=0.2 +trainer.limit_test_batches=0.2

Note: Visit configs/debug/ for different debugging configs.

python src/train.py ckpt_path="/path/to/ckpt/name.ckpt"

Note: Checkpoint can be either path or URL.

Note: Currently loading ckpt doesn't resume logger experiment, but it will be supported in future Lightning release.

# this will run 9 experiments one after the other,
# each with different combination of seed and learning rate
python src/train.py -m seed=100,200,300 model.optimizer.lr=0.0001,0.00005,0.00001

Note: Hydra composes configs lazily at job launch time. If you change code or configs after launching a job/sweep, the final composed configs might be impacted.

python src/train.py -m 'exp_maniskill2_act_policy/maniskill2_task@maniskill2_task=glob(*)'

Note: Hydra provides special syntax for controlling behavior of multiruns. Learn more here. The command above executes all task experiments from configs/exp_maniskill2_act_policy/maniskill2_task.

python src/train.py -m seed=100,200,300 trainer.deterministic=True

Note: trainer.deterministic=True makes pytorch more deterministic but impacts the performance.