computationalmodelling
FIX: Build Issue on MacOS
I'd like to give Fidimag a try, so I downloaded everything to my Mac and started to build it. Running the install scripts for Sundial and FTTW were fine, compiling using GCC 13 (installed via HomeBrew). But when I then went to build the main code set (using either 'make' or 'make build' from the root folder of the distribution), it got caught in an endless loop with the errors below; this chunk of compiler output repeated over and over again for hours.
Unsurprisingly, loading the system from within Python doesn't work. Exception Error message ----------------------- No module named 'fidimag.extensions.common_clib'
System Info
-----------------------
Architecture: arm64
Platform: macOS-13.5.1-arm64-arm-64bit
Processor: arm
Any suggestions on where to look for troubleshooting would be appreciated.
Output from "make":
Using CC=gcc-13
Building with 10 threads
Compiling /Users/dmsilev/mag_solve/fidimag/fidimag/atomistic/lib/clib.pyx because it changed.
Compiling /Users/dmsilev/mag_solve/fidimag/fidimag/common/lib/common_clib.pyx because it changed.
Compiling /Users/dmsilev/mag_solve/fidimag/fidimag/common/sundials/cvode.pyx because it changed.
Compiling /Users/dmsilev/mag_solve/fidimag/fidimag/micro/lib/baryakhtar/baryakhtar_clib.pyx because it changed.
Compiling /Users/dmsilev/mag_solve/fidimag/fidimag/micro/lib/micro_clib.pyx because it changed.
Compiling /Users/dmsilev/mag_solve/fidimag/fidimag/common/neb_method/nebm_clib.pyx because it changed.
Compiling /Users/dmsilev/mag_solve/fidimag/fidimag/common/dipolar/dipolar.pyx because it changed.
Compiling /Users/dmsilev/mag_solve/fidimag/fidimag/atomistic/fmmlib/fmm.pyx because it changed.
Traceback (most recent call last):
File "
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
Hi, from what I can see, the problem arises from building the C modules in parallel. Particularly from the spawn.py file.
Can you try to build Fidimag in Python 3.10? (or 3.9, where we know it works) You can make a new environment in conda:
conda create -n fidimag python=3.10
conda activate fidimag
From here, compile as usual. I'm not sure if the modules in 3.11 have changed too much. If this doesn't work we can try to check the compilation process in a macOS environment in Github Actions, and see if we can track the problem.
Also, be sure that you are using the system's GCC for both compiling Sundials and for building the modules (conda has a gcc compiler as well)
Hi David,
Thank you for the suggestions. I tried both a 3.9 and a 3.10 Python environment, and they both failed with the same issue. Just to be sure, I rebuilt both FFTW and Sundials in the new environments before running the overall make command. Since you mentioned that parallel processing might be an issue, I also tried editing setup.py to set nthreads=1 (rather than the 10 from counting cores on my machine); no luck there either, I'm afraid.
spawn.py does consistently seem to be the source of the problem.
Ok, thanks for clarifying. I was thinking the problem might be the arm64 architecture of your macOS machine, which is quite new.
I'll try to find a solution.
If you really want to try Fidimag, the only option for now might be to use Docker. There is a docker file in the main directory, which you can build using docker build -t fidimag . in the Dockerfile folder. You might want to change Ubuntu 18.04 by Ubuntu 20.04 in the script , as 18.04 is quite old but we know it works though.
This seems to be an open issue with cython on macOS https://github.com/NixOS/nixpkgs/issues/253205 :(
https://github.com/cython/cython/issues/3262
Can you try adding
if __name__ == "__main__":
multiprocessing.set_start_method('spawn', force=True)
to setup.py ?
Adding the set_start_method call doesn't seem to have changed anything.
I'll give the Docker approach a try when I have a chance.
Some progress. In setup.py, I changed the execute block to be this:
if __name__ == "__main__":
multiprocessing.set_start_method('spawn', force=True)
nthreads = multiprocessing.cpu_count()
print('Building with {} threads'.format(nthreads))
setup(
name='fidimag',
version=version,
description='Finite difference micromagnetic code',
packages=['fidimag',
'fidimag.atomistic',
'fidimag.micro',
'fidimag.extensions',
'fidimag.common',
],
ext_modules=cythonize(ext_modules, nthreads=nthreads),
)
i.e. wrapping the entire block inside the if main test. That resolved the infinite loop. I'm now getting a bunch of linker warnings and one error,
ld: warning: dylib (/Users/dmsilev/mag_solve/fidimag/local/lib/libsundials_nvecopenmp.dylib) was built for newer macOS version (13.0) than being linked (11.1) ld: warning: dylib (/opt/homebrew/Cellar/gcc/13.2.0/lib/gcc/current/libgomp.dylib) was built for newer macOS version (13.0) than being linked (11.1) ld: warning: dylib (/Users/dmsilev/mag_solve/fidimag/local/lib/libsundials_cvodes.dylib) was built for newer macOS version (13.0) than being linked (11.1) clang: error: unsupported option '-fopenmp' error: command '/usr/bin/clang++' failed with exit code 1
Something presumably was built with Clang rather than GCC-13 (the latter is specified with the CC environmental variable). I'll have to dig into that, but at least there's some hope.
Edit: The version warnings go away by setting an appropriate environmental variable: "export MACOSX_DEPLOYMENT_TARGET=13.0". Clang++ is still being called for some reason though.
Maybe https://apple.stackexchange.com/questions/245891/installed-gcc-with-homebrew-now-how-to-use-that-gcc-instead-of-clang
After following that suggestion and symlinking both cc and c++ to GCC, same issue. I verified that the symlinks work and Homebrew GCC is what should be called by default
(base) fidimag % c++ --version c++ (Homebrew GCC 13.2.0) 13.2.0
(base) fidimag % cc --version cc (Homebrew GCC 13.2.0) 13.2.0
and the shell Path variable is set correctly to see the Homebrew directory before /usr/bin, but something in the build routine is still trying to call /usr/bin/clang++. Going through the whole routine in a fresh conda environment (to try to make sure there weren't any stealth calls to Clang during the environmental setup process) didn't help either.
mm, interesting. The setup.py file searches for the compiler in these lines:
if 'CC' in os.environ:
print("Using CC={}".format(os.environ['CC']))
else:
os.environ["CC"] = "gcc"
print("Using CC={} (set by setup.py)".format(os.environ['CC']))
Maybe you need to set export CC=/usr/bin/gcc or so, but you already set that before.
Can you try adding these at the beginning?
os.environ["CC"] = "/usr/bin/gcc";
os.environ["CXX"] = "/usr/bin/g++"
(or wherever you have gcc and g++ installed)
Got it to build! To summarize for future reference...
Install GCC via Homebrew. Make two changes to setup.py. At the top, adding
os.environ["CC"] = "/usr/local/bin/gcc" os.environ["CXX"] = "/usr/local/bin/c++"
where gcc and c++ are symlinks to Homebrew (in my case /opt/homebrew/bin/gcc-13 and g++), and changing the bottom to be
if name == "main": multiprocessing.set_start_method('spawn', force=True) nthreads = multiprocessing.cpu_count() print('Building with {} threads'.format(nthreads)) setup( name='fidimag', version=version, description='Finite difference micromagnetic code', packages=['fidimag', 'fidimag.atomistic', 'fidimag.micro', 'fidimag.extensions', 'fidimag.common', ], ext_modules=cythonize(ext_modules, nthreads=nthreads), )
Two environmental variables set:
export CC = gcc-13 export MACOSX_DEPLOYMENT_TARGET=13.0
Thank you for all of the help and suggestions. Now to try actually using the software...
Awesome, I will probably add this to the documentation soon. Thanks to you for reporting the issue :D
It looks like there are still some issues, but now we're on to more subtle problems. For some reason, it's not picking up the LD_LIBRARY_PATH variable and when I run 'import fidimag' from within Python, it complains about not being able to find sundials_cvodes.2.dylib (which exists, as expected, in local/lib ).
If I just switch over to the local/lib directory and run Python from there, it then works, so that's enough for me to get started.
I see, maybe check https://stackoverflow.com/questions/3146274/is-it-ok-to-use-dyld-library-path-on-mac-os-x-and-whats-the-dynamic-library-s
And set the other related library paths: DYLD_LIBRARY_PATH in macOS
Also, be sure to not run fidimag from the main directory, as it will try to import the local Python libraries. Running it from a different directory, for instance, from the test or the other folders, is ok
Yeah, running it from the root folder seems to be the issue. If I run 'make test' from the root folder, it spams a whole bunch of 'can't find sundials' errors. Going to the tests folder and running python -m pytest instead did work correctly.