zhanweidu

icon indicating an email [email protected]

Results 3 comments of zhanweidu

NCCL communicator error: Socket Timeout when finetuning 70B model on 2 * (8* A100(80G))

set "CUDA_DEVICE_MAX_CONNECTIONS" to 32 maybe you need in environment. pls have a try @yguo33 @gonggaohan @tginart

Exception ignored in: <function AtomicFunction.__del__ at 0x71926a728940>

same problem for me! It seems the corrupt happened in tensorflow.python package, although I avoid using it as much as I can!

Hitting Container Kernel Memory Limit causes OOM on Docker Host

It seems the similar issue as we have met. We have installed k8s on our machine, and some of them always down because of oom killed by the docker daemon....