hollowgalaxy

in-cluster horovod tensorflow mnist example fails to build.

1

C++ version issue ``` Building: Step 14/24 : RUN ldconfig /usr/local/cuda-9.0/targets/x86_64-linux/lib/stubs && HOROVOD_GPU_ALLREDUCE=NCCL HOROVOD_WITH_TENSORFLOW=1 HOROVOD_WITH_PYTORCH=1 HOROVOD_WITH_MXNET=1 pip install --no-cache-dir horovod && ldconfig Building: Building: ---> Running in 74446ab9a86c Building: DEPRECATION:...

Currently get the error ``` file:///home/hollowgalaxy/images/output000075.png Could not open URL. Please check your internet connection and/or ssl settings If you are using proxy, make sure your proxy settings is configured...

I was attempting to have one of the models on 1 gpu, while the other model on another gpu, however, I get a crash. I was setting to dev_ids 0,...

My model isn't using any regularizers. Do I need weight decay? Or Should add l2? (Model in questions is a mix of 3d convolutions and 2d convolutions)