Single-node CPU-only performance is not good.

Open benson31 opened this issue 5 years ago • 2 comments

There have been reports of slow performance with default spack builds of LBANN in CPU-only mode. @denfromufa reported it in #1443 and I was able to verify in my local workstation build. For me, running the model_lenet_mnist.prototext model was showing about 75s/epoch with 1 OMP thread, 150s/epoch with 3 OMP threads. These are CUDA-less builds with Aluminum.

Feb 13 '20 23:02 benson31

I suspect this is just poor optimization. We haven't put much effort into CNNs on CPUs.

Feb 13 '20 23:02 timmoon10

That said, our LeNet integration test takes 5 sec/epoch on 2 Catalyst nodes. I wonder what's causing the difference.

Feb 13 '20 23:02 timmoon10