Naoto Usuyama comments

Results 44 comments of


                                            Naoto Usuyama

Using RDMA capable nodes

I also noticed NCCL_IB_DISABLE (env variable) is set to 1 by the pretrain AML environment (or maybe by the Docker image) ``` NCCL_IB_DISABLE The NCCL_IB_DISABLE variable disables the IB/RoCE transport...

Using RDMA capable nodes

When I tried the pretraining on ND24rs (RDMA/infiniband), I got the following error: ``` RuntimeError: NCCL error in: ... /torch/lib/c10d/ProcessGroupNCCL.cpp:290, unhandled system error ``` I think NCCL_IB_DISABLE should be set...

Using RDMA capable nodes

After checking with AzureML folks, it turned out I have to use Intel MPI as the backend when I use nodes without SR-IOV support. > SR-IOV stands for “single root...

docker folder doesn't exist for pretrain

From the name, this (https://github.com/Azure/AzureML-Containers/blob/master/base/gpu/openmpi3.1.2-cuda10.0-cudnn7-ubuntu16.04/Dockerfile ) looks like the base image; however, it seems there're additional steps for the AzureML-BERT image. Still looking for the Dockerfile.

validation data is missing

Duplicate #29 (still open)

Bert Data for Pretraining: No such file or directory: 'bert_data/validation_512_only'

For now I created `bert_data/validation_512_only` folder and moved `wikipedia_segmented_part_98.bin` and it seems the training pipeline is working fine. Still would be great to use the updated files @jingyanwangms

If input_img is a natural scene image?

U-net with image-net pretrained backbones should work well for natural scene images.

Use ConvTranspose2d instead

Thanks for a great suggestion! Please feel free to submit a PR.

RuntimeError with ResNetUNet

Hi @prarobinson thanks for trying this repo. Which version of PyTorch are you using? I created this repo with PyTorch 0.4.1 so there might be some breaking changes with new...

RuntimeError with ResNetUNet

Hmm that's interesting. Can you try to repro on an online notebook and share? e.g. https://colab.research.google.com/