Improvements for GPU binary wheels
This is a placeholder ticket to gather possible improvements for binary wheel:
- [ ] As noted here, see if
CMAKE_EXE_LINKER_FLAGSpollution could be avoided or worth doing. - [ ] Today if gpu wheel is installed and we try to use
nrnivmodl-core mod-dirwithout loading nvhpc module then we get compilation errors:
$ nrnivmodl-core .
INFO : Using neuron-gpu-nightly Package (Alpha Developer Version)
...
Translating x86_64/corenrn/mod2c/stim.mod into x86_64/corenrn/mod2c//stim.cpp
x86_64-linux-gnu-g++: error: unrecognized debug output level ‘pu=cuda11.0,cc60,cc70,cc80’
cvode_emit=0 cvode_not_allowed=0
Thread Safe
Translating x86_64/corenrn/mod2c/svclmp.mod into x86_64/corenrn/mod2c//svclmp.cpp
x86_64-linux-gnu-g++: error: unrecognized command line option ‘-mp’; did you mean ‘-p’?
x86_64-linux-gnu-g++: error: unrecognized command line option ‘--c++14’; did you mean ‘--stdc++14’?
cvode_emit=0 cvode_not_allowed=0
x86_64-linux-gnu-g++: error: unrecognized command line option ‘-acc’
Thread Safe
x86_64-linux-gnu-g++: error: unrecognized command line option ‘-Mautoinline’
make: *** [/home/kumbhar/nrn/nrn_test_venv_38/lib/python3.8/site-packages/neuron/.data/share/coreneuron/nrnivmodl_core_makefile:211: x86_64/corenrn/build/enginemech.o] Error 1
make: *** Waiting for unfinished jobs....
This is because nrnivmodl_core_makefile hardcodes PGI/NVHPC compiler flags. To avoid this, if GPU support is enabled then Makefile should check if CXX/CC is NVHPC compiler otherwise it should print an error and exit (e.g. "GPU enabled NEURON/CoreNEURON can be used with NVIDIA's NVHPC compiler. Please load module or set CC/CXX variables to nvc/nvc++ compilers")
- [x] Wheel should be built for which GPUs / Compute Capabilities? See comment here https://github.com/neuronsimulator/nrn/pull/1452#discussion_r732838876.
- [ ] Setup CI on BB5 to execute
packaging/python/test_wheels.shon CPU and GPU - [ ] Cray MPICH library is not libmpi.so but libmpich.so. Dynamic loading on NEURON side could be improved. (this is not specific to GPU though!)
https://bbpgitlab.epfl.ch/hpc/personal/neuron-gpu-wheel-testing is set up to run packaging/python/test_wheels.sh every night on a GPU node, but reporting the results is still work-in-progress.
https://github.com/neuronsimulator/nrn/pull/1657 makes the list compute capabilities explicit: 6.0, 7.0, 8.0. In https://github.com/BlueBrain/CoreNeuron/pull/713 the default changed to 7.0 and 8.0 because 6.0 is not supported by NVHPC with OpenMP target offload, but for now the GPU wheels use OpenACC (and mod2c).