EasyParallelLibrary
EasyParallelLibrary copied to clipboard
alibaba
Easy Parallel Library (EPL) is a general and efficient deep learning framework for distributed model training.
I got a problem when using EPL data parallel Model. The num worker is set to 3 and each worker had its own TF data record input and Model save_dir....
After I added below codes to my worked functions, I got a TypeError ```Python epl_config = epl.Config({ "gradient_checkpoint.type": "auto", "zero.level": "v1", "amp.level": "O1", "amp.loss_scale": 128 }) epl.init(epl_config) epl.set_default_strategy(epl.replicate(1)) ``` error...
**环境** nvcr.io/nvidia/tensorflow:21.12-tf1-py3镜像生成的容器 **代码：** FastNN/resnet/resnet_split.py **执行命令：** 服务器1：TF_CONFIG='{"cluster":{"worker":["172.20.21.181:55375","172.20.21.189:55376"]},"task":{"type":"worker","index":0}}' bash scripts/train_split.sh 服务器2：TF_CONFIG='{"cluster":{"worker":["172.20.21.181:55375","172.20.21.189:55376"]},"task":{"type":"worker","index":1}}' bash scripts/train_split.sh 服务器1的执行情况：  服务器2的执行情况：  可以看到服务器1的still waiting只打印了2条就不打印了说明已经接收到了服务器2的回复，但是没有继续往下运行。 **补充：** 同样的环境可以分布式运行bert，服务器之间是可以正常连接跑分布式训练的。 想问下是我的执行问题还是代码需要进行修改？
单机单卡： 启动命令：TF_CONFIG='{"cluster":{"worker":["127.0.0.1:49119"]},"task":{"type":"worker","index":0}}' CUDA_VISIBLE_DEVICES=0 bash ./scripts/train_dp.sh  单机双卡： 启动命令：TF_CONFIG='{"cluster":{"worker":["127.0.0.1:49119"]},"task":{"type":"worker","index":0}}' CUDA_VISIBLE_DEVICES=0,1 bash ./scripts/train_dp.sh  代码修改了一下：去掉了last_step限制，数据集repeat=10，将txt改为py，可执行。 [resnet_dp.txt](https://github.com/alibaba/EasyParallelLibrary/files/12671704/resnet_dp.txt) 想请教下，这个如何理解呢？每个卡分别跑了10step？
使用两个容器进行2机2卡实验，报错如下，希望可以帮忙解决一下 ### 环境: 基于nvcr.io/nvidia/tensorflow:21.12-tf1-py3构建的容器 ### 脚本: FastNN的resnet脚本 ### 启动命令 ``` TF_CONFIG='{"cluster":{"worker":["192.168.83.228:6666","192.168.83.228:6667"]},"task":{"type":"worker","index":0}}' bash scripts/train_dp.sh TF_CONFIG='{"cluster":{"worker":["192.168.83.228:6666","192.168.83.228:6667"]},"task":{"type":"worker","index":0}}' bash scripts/train_dp.sh ``` ### 报错 ``` 2023-08-31 01:40:46.786721: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at nccl_communicator.cc:116 : Internal:...
Hi EPL team, When I use epl library to train the following code: ```python import os import numpy as np from concurrent.futures import ThreadPoolExecutor from PIL import Image import tensorflow...
