MXNet.jl LoadError: Cannot find or load libmxnet.dylib

Hi, I have performed the following steps:

1). built MXNet from source with GPU support which generated the following 2 files:

libmxnet.a
libmxnet.so

2). withing Julia 0.6.2, Pkg.add("MXNet")

julia> Pkg.add("MXNet")
INFO: Installing MXNet v0.3.0
INFO: Building MXNet
INFO: MXNET_HOME environment detected: /Users/conradwt/ai.dir/projects/mxnet
INFO: Trying to load existing libmxnet...
INFO: Failed to load existing libmxnet, trying to build from source...
INFO: Found a CUDA installation.
INFO: Found a CuDNN installation.
INFO: CUDA_HOME -> /usr/local/cuda
INFO: Unsure if we can build against mkl.
INFO: Attempting build anyway.
INFO: USE_JULIA_BLAS -> true
INFO: Package database updated

3). within Julia 0.6.2, using MXNet

julia> using MXNet
INFO: Precompiling module MXNet.
ERROR: LoadError: LoadError: Cannot find or load libmxnet.dylib. Please see the document on how to build it.
Stacktrace:
 [1] error(::String) at ./error.jl:21
 [2] include_from_node1(::String) at ./loading.jl:576
 [3] include(::String) at ./sysimg.jl:14
 [4] include_from_node1(::String) at ./loading.jl:576
 [5] include(::String) at ./sysimg.jl:14
 [6] anonymous at ./<missing>:2
while loading /Applications/JuliaPro-0.6.2.2.app/Contents/Resources/pkgs-0.6.2.2/v0.6/MXNet/src/base.jl, in expression starting on line 38
while loading /Applications/JuliaPro-0.6.2.2.app/Contents/Resources/pkgs-0.6.2.2/v0.6/MXNet/src/MXNet.jl, in expression starting on line 25
ERROR: Failed to precompile MXNet to /Applications/JuliaPro-0.6.2.2.app/Contents/Resources/pkgs-0.6.2.2/lib/v0.6/MXNet.ji.
Stacktrace:
 [1] compilecache(::String) at ./loading.jl:710
 [2] _require(::Symbol) at ./loading.jl:497
 [3] require(::Symbol) at ./loading.jl:405

Note: libmxnet.dylib was not created but a libmxnet.so and libmxnet.a were created in step (1). Thus, I'm not sure why using MXNet is looking for a different file here.

Comment

Platform

macOS 10.13.3
Julia 0.6.2
MXNet 1.1.0

Mar 27 '18 04:03 conradwt

The build script will copy and rename it as libmxnet.dylib if build from Pkg.build.

https://github.com/dmlc/MXNet.jl/blob/3df1214e6b3e71d10d4f7156bcfb56e633b38f16/deps/build.jl#L255

Did you build your libmxnet manually? or via Pkg.build?

Mar 27 '18 16:03 iblislin

Also, the stable release of MXNet.jl might not work together with newer libmxnet. If you ran into issues, please try out master of MXNet.jl first. Lots of chages from stable release have been listed in NEWS.md.

Mar 27 '18 16:03 iblislin

I’m manually installing libmxnet from source from the project’s web site but I’ll try installing MXNet.jl using Pkg.build.

Mar 27 '18 20:03 conradwt

@iblis17 Please advise how to build MXNet.lj using Pkg.build. From the Julia documentation, there's no option for a single package. Finally, I believe part of the issue which I missed earlier is that Pkg.add("MXNet") is having trouble locating the package libmxnet.so.

Mar 27 '18 22:03 conradwt

In your case, I think adding a symblink (or just copy) libmxnet.so to libmxnet.dylib may work.

Please advise how to build MXNet.lj using Pkg.build.

try Pkg.build("MXNet")?

Finally, I believe part of the issue which I missed earlier is that Pkg.add("MXNet") is having trouble locating the package libmxnet.so.

yeah, I add libmxnet.so into search list for build script. Could you check out https://github.com/dmlc/MXNet.jl/pull/424 ?

Mar 28 '18 02:03 iblislin

I was able to resolve the Pkg.add("MXNet") by performing the following actions:

create a relative path within the libmxnet.so

$ install_name_tool -id @rpath/libmxnet.so libmxnet.so

create a symbolic link

$ ln -s libmxnet.so libmxnet.dylib

installing the package

julia> Pkg.add("MXNet")
INFO: Installing MXNet v0.3.0
INFO: Installing TakingBroadcastSeriously v0.1.1
INFO: Building MXNet
INFO: MXNET_HOME environment detected: /Users/conradwt/ai.dir/projects/mxnet
INFO: Trying to load existing libmxnet...
INFO: Existing libmxnet detected at /Users/conradwt/ai.dir/projects/mxnet/lib/libmxnet.dylib, skip building...
INFO: Found a CUDA installation.
INFO: Found a CuDNN installation.
INFO: CUDA_HOME -> /usr/local/cuda
INFO: Package database updated
INFO: METADATA is out-of-date — you may not have the latest version of MXNet
INFO: Use `Pkg.update()` to get the latest versions of your packages

using the package

julia> using MXNet
INFO: Recompiling stale cache file /Applications/JuliaPro-0.6.2.2.app/Contents/Resources/pkgs-0.6.2.2/lib/v0.6/MXNet.ji for module MXNet.

Using the install_name_tool and creating the symbolic link would be unnecessary if the appropriate build rules for macOS were added to Apache MXNet.

Mar 28 '18 12:03 conradwt

no idea why you need install_name_tool, in our Travis CI build, we simply do copy.

Mar 29 '18 03:03 iblislin

Does #424 works for you if not to make any symlink?

Mar 29 '18 03:03 iblislin

Adding just the sym-link didn’t work for me because this would generate a failure when I tried Pkg.add(“MXNet”). Why? The relative path, lib/libmxnet.so, within libmxnet.so isn’t correct. @rpath/libmxnet.so is a much better way to specify a relative path for shared and dynamic libraries.

Mar 29 '18 05:03 conradwt

Adding just the sym-link didn’t work for me because this would generate a failure when I tried Pkg.add(“MXNet”). Why? The relative path, lib/libmxnet.so, within libmxnet.so isn’t correct. .so is a much better way to specify a relative path for shared and dynamic libraries.

Sorry, I do not get detail issue you described. But I mean, don't make any symlink. Can just copying work for you?

Mar 29 '18 05:03 iblislin

If I don’t make a symlink, then I’ll get a failure.

Mar 29 '18 05:03 conradwt

Is it still a LoadError? mind share the error message?

or...could you try that copying your libmxnet.so to Pkg.dir("MXNet", "deps", "usr", "lib"), then rename it as libmxnet.dylib?

Mar 29 '18 06:03 iblislin

The error that I'm getting appear in my initial message which is as follows:

julia> Pkg.add("MXNet")
INFO: Installing MXNet v0.3.0
INFO: Building MXNet
INFO: MXNET_HOME environment detected: /Users/conradwt/ai.dir/projects/mxnet
INFO: Trying to load existing libmxnet...
INFO: Failed to load existing libmxnet, trying to build from source...
INFO: Found a CUDA installation.
INFO: Found a CuDNN installation.
INFO: CUDA_HOME -> /usr/local/cuda
INFO: Unsure if we can build against mkl.
INFO: Attempting build anyway.
INFO: USE_JULIA_BLAS -> true
INFO: Package database updated

Mar 29 '18 08:03 conradwt

Oh, I found the problem in my description. What I expected is this:

you already finished Pkg.add("MXNet")
cd to Pkg.dir("MXNet", "deps", "usr", "lib") (create this dir if not exists)
cp /path/to/your/libmxnet.so ./libmxnet.dylib
then, issue using MXNet in you REPL and check the precompilation success or not.

If you want stuffs work out of box from Pkg.add("MXNet") or Pkg.build("MXNet"), you should apply this patch first: https://github.com/dmlc/MXNet.jl/pull/424/files (hope this patch solve this Pkg.add issue).

Instructions: 1.

cd `julia -E 'Pkg.dir("MXNet")'`

git fetch origin pull/424/head:pr-424
git checkout pr-424
set your MXNET_HOME
Pkg.build("MXNet")
using MXNet

Mar 29 '18 09:03 iblislin

Here's the current status of things:

I copied the libmxnet.so to Pkg.dir("MXNet", "deps", "usr", "lib") and renamed it as libmxnet.dylib
downloaded this repo and switch to the v0.3.0 branch
julia lenet.jl ran successfully on the GPU.

Now, things appear to be working now. So, does the master branch have this patch? If so, how does one simply use the master branch? From the Julia documentation, one should be able to do the following: Pkg.checkout("MXNet"). However, this doesn't work as expected.

Mar 29 '18 09:03 conradwt

Now, things appear to be working now. So, does the master branch have this patch?

well, there are two situations:

without patch, just copying, and issue using MXNet.
applying patch, and run Pkg.build, using MXNet.

According to your operation in https://github.com/dmlc/MXNet.jl/issues/423#issuecomment-377176120. You chose approach 1 and it works.

So, does the master branch have this patch?

No, I sent it as a PR, not merged yet.

If so, how does one simply use the master branch? From the Julia documentation, one should be able to do the following: Pkg.checkout("MXNet"). However, this doesn't work as expected.

Pkg.checkout("MXNet") then Pkg.build("MXNet"). But note that master branch doesn't contain #424 .

Mar 29 '18 10:03 iblislin