cx_Oracle error for easyaccess

[hoyleb]

I'm on a mac OSX 10.11.4, and installed easyaccess using an up to date conda

$ python -V Python 2.7.12 :: Anaconda 4.0.0 (x86_64)

I keep getting this error message about Oracle. But I've not been able to fix it. I also got the error after installing via the "python install_oracleclient.py" script.

import easyaccess as ea File "//anaconda/lib/python2.7/site-packages/easyaccess.py", line 19, in import cx_Oracle File "//anaconda/lib/python2.7/site-packages/cx_Oracle-5.2.1-py2.7-macosx-10.5-x86_64.egg/cx_Oracle.py", line 7, in File "//anaconda/lib/python2.7/site-packages/cx_Oracle-5.2.1-py2.7-macosx-10.5-x86_64.egg/cx_Oracle.py", line 6, in bootstrap ImportError: dlopen(/Users/hoyleb/.python-eggs/cx_Oracle-5.2.1-py2.7-macosx-10.5-x86_64.egg-tmp/cx_Oracle.so, 2): Library not loaded: ./libclntsh.dylib.11.1 Referenced from: /Users/hoyleb/.python-eggs/cx_Oracle-5.2.1-py2.7-macosx-10.5-x86_64.egg-tmp/cx_Oracle.so Reason: image not found

[hoyleb]

I followed the advice here, and added my hostname to the /etc/hosts list. Now it seems to work okay!

http://chaos667.tumblr.com/post/20006357466/ora-21561-and-oracle-instant-client-112

[mgckind]

Thanks @hoyleb, I dropped the ball on this issue. Isn't the same problem we talked here: https://opensource.ncsa.illinois.edu/confluence/display/DESDM/questions/78874410/problem-with-connection-to-the-database-oid

It seems that the host 'trick' fixes several Oracle errors !

I'll go ahead an close this issue

[kadrlica]

I just ran into this too trying a conda install on OS X 10.9.5:

Traceback (most recent call last):
  File "/usr/local/anaconda2/envs/ea/bin/easyaccess", line 19, in <module>
    import cx_Oracle
  File "build/bdist.macosx-10.5-x86_64/egg/cx_Oracle.py", line 7, in <module>
  File "build/bdist.macosx-10.5-x86_64/egg/cx_Oracle.py", line 6, in __bootstrap__
ImportError: dlopen(/Users/alex/Library/Caches/Python-Eggs/cx_Oracle-5.2.1-py2.7-macosx-10.5-x86_64.egg-tmp/cx_Oracle.so, 2): Library not loaded: ./libclntsh.dylib.11.1
  Referenced from: /Users/alex/Library/Caches/Python-Eggs/cx_Oracle-5.2.1-py2.7-macosx-10.5-x86_64.egg-tmp/cx_Oracle.so
  Reason: image not found

Editing /etc/hosts didn't work for me. I have a version of cx_Oracle that I installed by hand, but I was hoping to switch to the conda install.

[mgckind]

Thanks! I recently saw a similar issue with somebody else, not sure the reason for this. This other person was using python3 and a latest OS. Not sure whether is a library missing issue or something else

Need to understand this better before cutting the 1.3.2 release :|. Will reopen this issue

[kadrlica]

I think the library (libclntsh.dylib.11.1) exists in the environment lib directory:

>ls /usr/local/anaconda2/envs/ea/lib/libclntsh.dylib.11.1
/usr/local/anaconda2/envs/ea/lib/libclntsh.dylib.11.1

but it is not linked (or copied or whatever) to the tmp directory that oracle is creating:

>ls /Users/alex/Library/Caches/Python-Eggs/cx_Oracle-5.2.1-py2.7-macosx-10.5-x86_64.egg-tmp/
cx_Oracle.so

Because anaconda is not setting the LD_LIBRARY_PATH (or whatever), it can't find the shared library in the local directory where it is looking. I have no idea why oracle (or maybe python) is creating this egg-tmp directory, but that is where I would look first.

[kadrlica]

Ok, by specifically adding the environment lib directory to the library path I've gotten things to work:

> conda create -n ea easyaccess=1.3.1
> source activate ea
> export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:$(conda info --root)/envs/$CONDA_DEFAULT_ENV/lib
> easyaccess

I think that this means something might have gone wrong with the dynamic linking when easyaccess (or more likely cx_oracle) was built as a conda package. The conda documentation explicitly discourages setting the (DY)LD_LIBRARY_PATH variables, so the solution above is probably not sustainable.

[kadrlica]

Ok, I think that the issue is in the relative linked path to ./libclntsh.dylib.11.1 whereas this link should be @rpath/libclntsh.dylib.11.1.

For mgckind/cx_oracle I see that:

> otool -L /Users/alex/Library/Caches/Python-Eggs/cx_Oracle-5.2.1-py2.7-macosx-10.5-x86_64.egg-tmp/cx_Oracle.so
/Users/alex/Library/Caches/Python-Eggs/cx_Oracle-5.2.1-py2.7-macosx-10.5-x86_64.egg-tmp/cx_Oracle.so:
    ./libclntsh.dylib.11.1 (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

Whereas, if I look at jsandhu/cx_oracle:

> otool -L /usr/local/anaconda2/pkgs/cx_oracle-5.2.1-py27_4/lib/python2.7/site-packages/cx_Oracle-5.2.1-py2.7-macosx-10.5-x86_64.egg/cx_Oracle.so 
/usr/local/anaconda2/pkgs/cx_oracle-5.2.1-py27_4/lib/python2.7/site-packages/cx_Oracle-5.2.1-py2.7-macosx-10.5-x86_64.egg/cx_Oracle.so:
    @rpath/libclntsh.dylib.11.1 (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

(however jsandhu/cx_oracle has it's own set of install problems.)

I'm not sure yet how to impose this relative linking when building cx_oracle.

[kadrlica]

Might want to take a look at the build here: https://anaconda.org/jsandhu/cx_oracle/5.2.1/download/osx-64/cx_oracle-5.2.1-py27_4.tar.bz2

Specifically, in the README:

#!/bin/bash

#
# Have to manually fix the build and repackage
#

NAME=cx_oracle-5.2.1-py27_1
TNAME=$NAME'.tar'
BNAME=$NAME'.tar.bz2'
ENAME=cx_Oracle-5.2.1.egg

bunzip2 $BNAME
mkdir tmp
cp $TNAME tmp/
cd tmp
tar xvf $TNAME

cd lib/python*/site-packages/
mkdir $ENAME && mv cx_Oracle*.egg $ENAME 
cd $ENAME && unzip *.egg
rm *.egg

install_name_tool -add_rpath @loader_path/../../../ cx_Oracle.so

cd ../../../../
rm $TNAME
tar cvf $TNAME info/ lib/
bzip2 -z $TNAME
mv $BNAME ../.

# Also update the contents of info/

However, I have trouble installing jsandhu/cx_oracle, I think because the egg is a directory and conda is failing to symlink.

Linking packages ...
CondaOSError: OS error: failed to link (src=u'/usr/local/anaconda2/pkgs/cx_oracle-5.2.1-py27_4/lib/python2.7/site-packages/cx_Oracle-5.2.1-py2.7-macosx-10.5-x86_64.egg/', dst='/usr/local/anaconda2/envs/ea2/lib/python2.7/site-packages/cx_Oracle-5.2.1-py2.7-macosx-10.5-x86_64.egg/', type=1, error=OSError(17, 'File exists'))

[kadrlica]

Ok, I think things go deeper than cx_Oracle into the instant client. It looks like we need to set the library dependencies to use relative paths. This is a complicated issue, but if you look at the oracle-instantclient libraries, you'll see that they have local relative paths :

> otool -L libclntsh.dylib.11.1
libclntsh.dylib.11.1:
    ./libclntsh.dylib.11.1 (compatibility version 0.0.0, current version 0.0.0)
    ./libnnz11.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)

otool -L libnnz11.dylib
libnnz11.dylib:
    ./libnnz11.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)

...

What needs to happen is that these local relative paths (i.e., paths starting with ./) need to be converted to @rpath values. You can find more details on conda issue #279. As an (mostly) working example, you can take a look at the version of the instant client built by jsandhu.

I think I'm getting pretty close to getting things working, but it will require a re-build of the instant client and of cx_Oracle.

[kadrlica]

Ok, it appears that I've got it working. So the kicker was to use jsandhu/oracle-instantclient.

After that switch (which updates all of the instantclient libraries to use @rpath) the build and installation of cx_Oracle are "reasonably" straight forward.

meta.yaml

package:
  name: cx_oracle
  version: "5.2.1"

source:
  fn: cx_Oracle-5.2.1.tar.gz
  url: https://pypi.python.org/packages/source/c/cx_Oracle/cx_Oracle-5.2.1.tar.gz
  md5: 65a6bcc5217a9502c10e33fcea2982f3

build:
  preserve_egg_dir: True
  conda_binary_relocation: True
  number: 0

requirements:
  build:
    - python
    - setuptools
    - oracle-instantclient # This needs to come from jsandhu

  run:
    - python
    - oracle-instantclient # This needs to come from jsandhu

test:
  # Python imports
  imports:
    - cx_Oracle

about:
  home: http://cx-oracle.sourceforge.net
  license: Python Software Foundation License
  summary: 'Python interface to Oracle'

# See
# http://docs.continuum.io/conda/build.html for
# more information about meta.yaml

build.sh

#!/bin/bash

export ORACLE_HOME=$CONDA_PREFIX/lib
export FORCE_RPATH=1

$PYTHON setup.py install

[kadrlica]

I've copied jsandhu/oracle-instantclient and rebuilt my own version of cx_oracle. The following now works for me to install easyaccess:

> conda create -n ea easyaccess=1.3.1 -c kadrlica -c mgckind

It would be good to put everything in one place, but I think that you (Matias) can just copy the version of oracle-instantclient and cx_oracle:

> anaconda copy jsandhu/oracle-instantclient/11.2.0.4.0
> anaconda copy kadrlica/cx_oracle/5.2.1

(I'm not sure how anaconda copy resolves issues with conflicting package names.)

[mgckind]

Thanks a lot! @kadrlica It was a busy weekend but I've followed this closely. I hope to have the 1.3.2 version ready this week and I'll definitely use this in the new conda version,

[kadrlica]

Great! Please test and let me know if there are any problems.

Be warned about about anaconda copy. I ran into some issues when trying to copy to pre-existing packages (#355). I'd recommend installing the new packages from my channel and then uploading them to your own channel.

[ricardogando]

Great, @kadrlica . Thanks for this debugging. I had this issue too with 10.12.1 (Sierra) and now it is working. Will only add to casual conda user how to setup ea environment. On home run: source activate ea

[mgckind]

Thanks Ricardo,

Version 1.4.0 is basically ready, the only pending is to create conda packages to solve these issues

[kadrlica]

Anything I can help with for 1.4.0?