Does it support model parallel? How to convert ColumnParallelLinear to lora linear?

[LiuShixing]

how to convert Megatron-DeepSpeed ColumnParallelLinear and RowParallelLinear to lora linear layer?

ColumnParallelLinear defined in: https://github.com/microsoft/Megatron-DeepSpeed/blob/main/megatron/mpu/layers.py#L206

[edwardjhu]

Hi LiuShixing,

My suggestion is to implement your version of LoRA for this instead of using what I wrote in this library, which is intended to replace basic layers such as nn.Linear. You can use my code as an example of how to translate the idea in our paper into code.

[LiuShixing]

Think you, I had implement it

[BrightXiaoHan]

@LiuShixing 你好，我最近也想将lora适配到模型并行上面去，方便加微信交流一下么，15637899910

[wsh2836741]

@LiuShixing @BrightXiaoHan Hi，could you share your code? Thanks very much!