OpenDiT icon indicating copy to clipboard operation
OpenDiT copied to clipboard
verified publisher icon NUS-HPC-AI-Lab

[Question] Do you compare FastSeq with RingAttention or Context Parallel in TransformerEngine?

Open baoleai opened this issue 2 years ago • 1 comments

This is an impressive project! It demonstrates superior performance compared to Ulysses. Have you also conducted comparisons of FastSeq with RingAttention or Context Parallel in the TransformerEngine?

baoleai avatar Feb 28 '24 06:02 baoleai

We will be updating our resuilts with more baselines soon

oahzxl avatar Feb 28 '24 07:02 oahzxl