Li Zhu

@little51 How to support multi-GPU training？ > The flash_attn is not supported.Use load_in_8bit,peft technology and bitsandbytes to accelerate.It requires about 13G of GPU memory. > > https://github.com/git-cloner/llama-lora-fine-tuning#341-fine-tuning-command for the training...

> @zl1994 Multi gpu will encounter RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu, which has not been resolved...

@gaosanyuan In the last train_maskrcnn stage, it was initialized with rpn2+rcnn1. But I found it actually be initialized with rcnn1 for the rpn2's feature extraction part was initialized with rcnn1...