CoreNeuron
CoreNeuron copied to clipboard
Issue with OpenMP enabled python binary wheels
Describe the issue
Like OpenACC/CUDA, ABI compatibility of OpenMP and shipping duplicate OpenMP runtime library is problematic.
To Reproduce
- Build the wheel on manylinux2014_aarch64
- Try to use the wheel on Ubuntu20 Aarch64 (circle-ci) gives:
(nrn_test_venv_39) circleci@ip-172-28-90-45:~/project$ python test/coreneuron/test_direct.py
NEURON: unable to open font "*helvetica-medium-r-normal*--14*", using "fixed"
/lib/aarch64-linux-gnu/libgomp.so.1: cannot allocate memory in static TLS block
NEURON: Could not dlopen CoreNEURON mechanism library : /home/circleci/project/aarch64/libcorenrnmech.so
near line 0
create soma
^
ParallelContext[0].psolve(5)
Traceback (most recent call last):
File "/home/circleci/project/test/coreneuron/test_direct.py", line 92, in <module>
test_direct_memory_transfer()
File "/home/circleci/project/test/coreneuron/test_direct.py", line 70, in test_direct_memory_transfer
run(mode)
File "/home/circleci/project/test/coreneuron/test_direct.py", line 52, in run
pc.psolve(h.tstop)
RuntimeError: hocobj_call error
Expected behavior
Wheel should work without any issue.
Additional context
- See relevant discussion in https://github.com/opencv/opencv/issues/14884 and OpenMP related
-
auditwheelpackages extra libraries like OpenMP
$ ldd aarch64/libcorenrnmech.so
linux-vdso.so.1 (0x0000ffffb379e000)
libcoreneuron.so => /home/circleci/project/nrn_test_venv_39/lib/python3.9/site-packages/neuron/.data/lib/libcoreneuron.so (0x0000ffffb35ce000)
libstdc++.so.6 => /lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000ffffb33dc000)
libm.so.6 => /lib/aarch64-linux-gnu/atomics/libm.so.6 (0x0000ffffb332e000)
libgomp.so.1 => /lib/aarch64-linux-gnu/libgomp.so.1 (0x0000ffffb32e0000)
libc.so.6 => /lib/aarch64-linux-gnu/atomics/libc.so.6 (0x0000ffffb316f000)
/lib/ld-linux-aarch64.so.1 (0x0000ffffb376e000)
libdl.so.2 => /lib/aarch64-linux-gnu/atomics/libdl.so.2 (0x0000ffffb315b000)
libgomp-d22c30c5.so.1.0.0 => /home/circleci/project/nrn_test_venv_39/lib/python3.9/site-packages/neuron/.data/lib/../../../NEURON_nightly.libs/libgomp-d22c30c5.so.1.0.0 (0x0000ffffb3107000)
libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000ffffb30e2000)
libpthread.so.0 => /lib/aarch64-linux-gnu/atomics/libpthread.so.0 (0x0000ffffb30b3000)
On the host machine OpenMP runtime exists in:
(nrn_test_venv_39) circleci@ip-172-28-90-45:~/project$ ls /usr/lib/aarch64-linux-gnu/libgomp*
/usr/lib/aarch64-linux-gnu/libgomp.so.1 /usr/lib/aarch64-linux-gnu/libgomp.so.1.0.0
And forcing this library makes thing "better":
(nrn_test_venv_39) circleci@ip-172-28-90-45:~/project$ LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libgomp.so.1 python test/coreneuron/test_direct.py
NEURON: unable to open font "*helvetica-medium-r-normal*--14*", using "fixed"
================ MECHANISMS COUNT BY TYPE ==================
Id Name Count
0 (null) 0
1 (null) 0
2 morphology 0
3 capacitance 1
4 pas 0
5 extracellular 0
6 fastpas 0
7 IClamp 1
8 AlphaSynapse 0
9 ExpSyn 0
10 Exp2Syn 0
11 SEClamp 0
12 VClamp 0
13 OClamp 0
14 APCount 0
15 na_ion 1
16 k_ion 1
17 hh 1
18 NetStim 0
19 IntFire1 0
20 IntFire2 0
21 IntFire4 0
22 PointProcessMark 0
23 PatternStim 0
24 ba1 0
25 ba0 0
26 ForNetConTest 0
27 IntervalFire 0
28 DAsyn 0
29 Sample 1
30 Bounce 0
=============================================================
Memory size information for all NrnThreads per rank
------------------------------------------------------------------
field min max avg
n_cell 1 1 1.00
....
@ferdonline : we have similar issue with GPU (OpenACC runtime) and you can see @olupton's hack in https://github.com/neuronsimulator/nrn/pull/1661/files. One of previous discussion was if we should workaround this in auditwheel.
For reference: https://github.com/pypa/auditwheel/pull/368