Number of Gpus used for pre-training

Open 6Roy opened this issue 3 years ago • 2 comments

Hi, it is mentioned in the paper that you pretrain the model on two 16-GPU nodes, I wonder is it using 16 A100 gpus? I'm not sure what this nodes means？ Thanks a lot.

Feb 16 '23 14:02 6Roy

it is 32 A100 in total.

Feb 20 '23 00:02 LiJunnan1992

Typical GPU machines have up to 8 GPUs. NVIDIA DGX-2 seems to have 16 GPUs (but with V100). I am not aware of any computer with 16 A100 GPUs. I got DGX configs from https://en.wikipedia.org/wiki/Nvidia_DGX .

Apr 23 '23 17:04 youkaichao