The training takes forever

[Lwy-1998]

I set the N_rand to 384 and chunk_size to 512, to make sure it can run on my rtx3090. But the training process is very slow,is there any way to accelerate it?

[zhengqili]

Hi, Unfortunately current method only works well for large N_rand (> 2500 in my opinion), which requires at least 4x 30G GPUs for training usage.