CoreNeuron icon indicating copy to clipboard operation
CoreNeuron copied to clipboard

Improvements for GPU binary wheels

Open pramodk opened this issue 4 years ago • 2 comments

This is a placeholder ticket to gather possible improvements for binary wheel:

  • [ ] As noted here, see if CMAKE_EXE_LINKER_FLAGS pollution could be avoided or worth doing.
  • [ ] Today if gpu wheel is installed and we try to use nrnivmodl-core mod-dir without loading nvhpc module then we get compilation errors:
$ nrnivmodl-core .
INFO : Using neuron-gpu-nightly Package (Alpha Developer Version)
...
Translating x86_64/corenrn/mod2c/stim.mod into x86_64/corenrn/mod2c//stim.cpp
x86_64-linux-gnu-g++: error: unrecognized debug output level ‘pu=cuda11.0,cc60,cc70,cc80’
cvode_emit=0 cvode_not_allowed=0
Thread Safe
Translating x86_64/corenrn/mod2c/svclmp.mod into x86_64/corenrn/mod2c//svclmp.cpp
x86_64-linux-gnu-g++: error: unrecognized command line option ‘-mp’; did you mean ‘-p’?
x86_64-linux-gnu-g++: error: unrecognized command line option ‘--c++14’; did you mean ‘--stdc++14’?
cvode_emit=0 cvode_not_allowed=0
x86_64-linux-gnu-g++: error: unrecognized command line option ‘-acc’
Thread Safe
x86_64-linux-gnu-g++: error: unrecognized command line option ‘-Mautoinline’
make: *** [/home/kumbhar/nrn/nrn_test_venv_38/lib/python3.8/site-packages/neuron/.data/share/coreneuron/nrnivmodl_core_makefile:211: x86_64/corenrn/build/enginemech.o] Error 1
make: *** Waiting for unfinished jobs....

This is because nrnivmodl_core_makefile hardcodes PGI/NVHPC compiler flags. To avoid this, if GPU support is enabled then Makefile should check if CXX/CC is NVHPC compiler otherwise it should print an error and exit (e.g. "GPU enabled NEURON/CoreNEURON can be used with NVIDIA's NVHPC compiler. Please load module or set CC/CXX variables to nvc/nvc++ compilers")

  • [x] Wheel should be built for which GPUs / Compute Capabilities? See comment here https://github.com/neuronsimulator/nrn/pull/1452#discussion_r732838876.
  • [ ] Setup CI on BB5 to execute packaging/python/test_wheels.sh on CPU and GPU
  • [ ] Cray MPICH library is not libmpi.so but libmpich.so. Dynamic loading on NEURON side could be improved. (this is not specific to GPU though!)

pramodk avatar Oct 20 '21 21:10 pramodk

https://bbpgitlab.epfl.ch/hpc/personal/neuron-gpu-wheel-testing is set up to run packaging/python/test_wheels.sh every night on a GPU node, but reporting the results is still work-in-progress.

olupton avatar Feb 24 '22 17:02 olupton

https://github.com/neuronsimulator/nrn/pull/1657 makes the list compute capabilities explicit: 6.0, 7.0, 8.0. In https://github.com/BlueBrain/CoreNeuron/pull/713 the default changed to 7.0 and 8.0 because 6.0 is not supported by NVHPC with OpenMP target offload, but for now the GPU wheels use OpenACC (and mod2c).

olupton avatar Feb 24 '22 17:02 olupton