CTranslate2 icon indicating copy to clipboard operation
CTranslate2 copied to clipboard
verified publisher icon OpenNMT

tensor parallel by nccl + mpi

Open minhthuc2502 opened this issue 2 years ago • 1 comments

WIP for the feature tensor parallel. There are some points to investigate:

  • Make new version of converter to move forward the number heads before the appearance of weight, bias in self attention to deal with group query attention.
  • Packaging python wrapper: how to deal with MPI and NCCL when packaging

minhthuc2502 avatar Jan 11 '24 16:01 minhthuc2502

LGTM. It helps me a lot. I'm looking forward to seeing the full release version.

duydq12 avatar Jan 12 '24 07:01 duydq12