Install fails in [email protected] with Anaconda environment.

[mortvif]

I'm trying to install PyNode 0.5.1 (last version available in npm) in my system using my usual anaconda environment to execute the python code. The versions currently used in my system are as follows:

Ubuntu 20.04.1 with gcc 9.3.0 Python 3.6 Conda 4.9.1 with Python 3.6.12 installed Node 12.19 node-gyp 5.1.0

However, I am facing an error while compiling the package. It seems that some packages are expected to be built with a different toolchain version (see attached log). Do you have any idea on how to solve this issue?

Thanks!

> @fridgerator/[email protected] install /home/sandra/data-parser/node_modules/@fridgerator/pynode
> node-gyp rebuild

make: se entra en el directorio '/home/sandra/data-parser/node_modules/@fridgerator/pynode/build'
  CC(target) Release/obj.target/nothing/../../node-addon-api/src/nothing.o
  AR(target) Release/obj.target/../../node-addon-api/src/nothing.a
  COPY Release/nothing.a
  CXX(target) Release/obj.target/PyNode/src/main.o
  CXX(target) Release/obj.target/PyNode/src/helpers.o
  CXX(target) Release/obj.target/PyNode/src/pynode.o
  CXX(target) Release/obj.target/PyNode/src/worker.o
  CXX(target) Release/obj.target/PyNode/src/pywrapper.o
  CC(target) Release/obj.target/PyNode/src/jswrapper.o
  SOLINK_MODULE(target) Release/obj.target/PyNode.node
lto1: fatal error: bytecode stream in file ‘/home/sandra/anaconda3/lib/python3.6/config-3.6m-x86_64-linux-gnu/libpython3.6m.a’ generated with LTO version 6.0 instead of the expected 8.1
compilation terminated.
lto-wrapper: fatal error: g++ returned 1 exit status
compilation terminated.
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
make: *** [PyNode.target.mk:161: Release/obj.target/PyNode.node] Error 1
make: se sale del directorio '/home/sandra/data-parser/node_modules/@fridgerator/pynode/build'
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (events.js:314:20)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:275:12)
gyp ERR! System Linux 5.4.0-51-generic
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/sandra/data-parser/node_modules/@fridgerator/pynode
gyp ERR! node -v v12.19.0
gyp ERR! node-gyp -v v5.1.0
gyp ERR! not ok
npm WARN [email protected] No repository field.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! @fridgerator/[email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the @fridgerator/[email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/sandra/.npm/_logs/2020-10-28T11_54_45_947Z-debug.log

[fridgerator]

Just looking up the error "generated with LTO version 6.0 instead of the expected 8.1" seems to be somewhat common. Seems like possibly an outdated gcc version?

[mortvif]

It do seems to be a problem with gcc versions. Aparently, the python version used within the conda environment has been compiled with gcc version 7.3 wheras the default gcc version on Ubuntu 20.04 is 9.3. Unfortunately, 7.3 version is not available anymore on Ubuntu 20.04.

Therefore, I am still unable to use pynode and the conda environment at the same time. I have avoided this problem by using the "native" (non-conda) python version which is compiled using gcc-v9.3. In this case, I can install and use Pynode without any issues.

[milliele]

I just succeeded in using PyNode with Miniconda on Ubuntu 18.04. Just to share how I finally make it:

TL; DR

1. Installed newer python: conda create -n py38 python=3.8, conda activate py38
2. Installed required python packages: pip install gyp-next find_libpython
3. Find path of python shared library: find_libpython or python -m find_libpython. Suppose it be /xxx/miniconda3/envs/py38/lib/libpython3.8.so.1.0.
4. Install PyNode: PYTHON_SHARED=/xxx/miniconda3/lib PY_LIBS="$(python ../build_ldflags.py) -L$PYTHON_SHARED -Wl,-rpath=$PYTHON_SHARED" npm install @fridgerator/pynode
5. If you got an error like undefined symbol: PyLong_AsLongLongAndOverflow when importing some python module, just use pynode.dlOpen(path) to link to this shared library again with path=/xxx/miniconda3/envs/py38/lib/libpython3.8.so.1.0 (i.e. the output of find_libpython).

Details

At first, I encountered the lto1 error as well:

lto1: fatal error: bytecode stream in file ‘/home/sandra/anaconda3/lib/python3.6/config-3.6m-x86_64-linux-gnu/libpython3.6m.a’ generated with LTO version 6.0 instead of the expected 8.1

I believe this is because the python installed and currently used on Conda is built by outdated GCC version as @fridgerator has mentioned.

Then I installed another python environment in Conda with a newer version:

conda create -n py38 python=3.8
(restart shell)
conda activate py38

Then you might find an error that -llibpython3.8 is not found:

/usr/bin/ld: cannot find -lpython3.8

This is because the python executable installed by conda is built by a static library like libpython3.8.a rather than a shared library. If you check the dynamic library dependency of your python:

(py38) $ ldd `which python`

linux-vdso.so.1 (0x00007ffe03efd000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007faa187be000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007faa185bb000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007faa1821d000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007faa17ffe000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007faa17c0d000)
/lib64/ld-linux-x86-64.so.2 (0x00007faa18d5d000)

There is no such thing as libpython3.8.so, that's why we can't find -lpython3.8. What we need to do is to add the directory where libpython3.8.so exists in the LDFLAGS, by adding -L<path_to_lib> to environment variable PY_LIBS leveraged by PyNode.

Fortunately, (1) conda still provides the dynamic library, and (2) there is a python package find_libpython that could help us find the path.

(py38) $ pip install find_libpython
(py38) $ find_libpython

/xxx/miniconda3/envs/py38/lib/libpython3.8.so.1.0

Then let PY_LIBS be the concatenation of the directory path and the output of python build_ldflags.py:

(py38) $ PY_LIBS="$(python build_ldflags.py) -L/xxx/miniconda3/envs/py38/lib" npm install @fridgerator/pynode

Thus I installed PyNode successfully. However node.js still failed to find the shared library when it executing the PyNode addon, and we will get:

(py38) $ node
> const pynode = require('@fridgerator/pynode');

Uncaught:
Error: libpython3.8.so.1.0: cannot open shared object file: No such file or directory
    at Object.Module._extensions..node (node:internal/modules/cjs/loader:1168:18)
    at Module.load (node:internal/modules/cjs/loader:989:32)
    at Function.Module._load (node:internal/modules/cjs/loader:829:14)
    at Module.require (node:internal/modules/cjs/loader:1013:19)
    at require (node:internal/modules/cjs/helpers:93:18) {
  code: 'ERR_DLOPEN_FAILED'
}

(py38) $ ldd node_modules/@fridgerator/pynode/build/Release/PyNode.node

linux-vdso.so.1 (0x00007ffce1781000)
libpython3.8.so.1.0 => not found
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd675499000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fd675110000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fd674ef8000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd674b07000)
/lib64/ld-linux-x86-64.so.2 (0x00007fd6758ba000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd674769000)

We need to add a -Wl,-rpath option to tell the path to the directory like:

-Wl,-rpath=/xxx/miniconda3/envs/py38/lib

, which means actually we need to install PyNode like:

(py38) $ PYTHON_SHARED=/xxx/miniconda3/lib PY_LIBS="$(python ../build_ldflags.py) -L$PYTHON_SHARED -Wl,-rpath=$PYTHON_SHARED" npm install @fridgerator/pynode

And thus pynode would work well:

(py38) $ ldd node_modules/@fridgerator/pynode/build/Release/PyNode.node

	linux-vdso.so.1 (0x00007ffca67dc000)
	libpython3.8.so.1.0 => /xxx/miniconda3/envs/py38/lib/libpython3.8.so.1.0 (0x00007f2b384c2000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2b382be000)
	libstdc++.so.6 => /mnt/asp_test/env/miniconda3/lib/libstdc++.so.6 (0x00007f2b38b23000)
	libgcc_s.so.1 => /mnt/asp_test/env/miniconda3/lib/libgcc_s.so.1 (0x00007f2b38b0f000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2b37ecd000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f2b37cca000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2b3792c000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2b3770d000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f2b38a82000)
	
(py38) $ node
> const pynode = require('@fridgerator/pynode');
undefined

If you got an error like undefined symbol: PyLong_AsLongLongAndOverflow when importing some python module, just use pynode.dlOpen(path) to link to this shared library again with path=/xxx/miniconda3/envs/py38/lib/libpython3.8.so.1.0 (i.e. the output of find_libpython).

[txchen]

@milliele Thanks for the write up! May I know the content of build_ldflags.py you mentioned?

[milliele]

@milliele Thanks for the write up! May I know the content of build_ldflags.py you mentioned?

I didn't make any changes to the files. It's just this file in the repo: https://github.com/fridgerator/PyNode/blob/master/build_ldflags.py.