GCC with -Ofast must be linked to libm for some kernels

[michalhabera]

Some kernels need to call CFFI with libraries=["m"] kwarg when compiling with GCC and -Ofast optimization level. I suppose -Ofast is doing some transformation which implicitly assume linkage with math.h.

[chrisrichardson]

@michalhabera - would it be safe to always add -lm linkage? I think the linker will just ignore if not used.

[garth-wells]

Is this because we include math.h in ufc_geometry.h?

[michalhabera]

I only observe this when using -Ofast, i.e. -ffast-math. For -O3 no missing symbol issues. The typical error is ImportError: /root/.cache/fenics/libffcx_forms_ef6d1600eef81c98b3d6319ebc286a08d493b291.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZGVdN4v_exp.

_ZGVdN4v_exp is a SIMD version of exp(), so the compiler did some transform which could benefit from vectorised exp, but it was not linked.

[garth-wells]

@michalhabera can you post the UFL file? If exp is being called we should be including math.h in the generated code and the FFCX JIT should know that the library m needs to be linked against.

[michalhabera]

Is linking against libm always safe? Maybe we can add it as a default, any experienced user would link against his math library himself.

[michalhabera]

This is still an issue. The problems comes from missing linkage to libmvec.so. We always include math.h, but we never link against libm.so explicitly when we compile the kernel. However, in most cases this just works, because python links against libm.so itself (at least on my system, not sure if guaranteed on all),

root@7f816068061f:/home/tests# ldd /usr/bin/python3
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fffff0ea000)
	libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007fffff0b9000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fffff09d000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffffee74000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ffffffc4000)

The situation changes when compiler emits vectorized math calls, such as _ZGVbN2v_sin, _ZGVdN4v_exp, .... Compiler/linker knows, that if the binary links against libm and there are vectorized math function, it has to add libmvec linkage implicitly (https://sourceware.org/glibc/wiki/libmvec). But since we rely on existing linkage of python, no libmvec is being added.

Here is small code that fails if run on amd64 target (I could not get my Apple M1 emit vectorized sin call, so had to run in amd64 container)

import dolfinx
from mpi4py import MPI
import ufl

mesh = dolfinx.mesh.create_unit_square(MPI.COMM_WORLD, 10, 10)

ufl_form = ufl.sin(ufl.SpatialCoordinate(mesh)[0])*ufl.dx
form = dolfinx.fem.form(ufl_form, jit_options={"cffi_extra_compile_args":["-Ofast", "-march=native"],
                                               "cffi_verbose": True, "cffi_libraries": []})

because obviously

root@7f816068061f:/home/tests# ldd /root/.cache/fenics/libffcx_forms_1983535cfbcc858b2908aa5664d28236c046a33f.cpython-310-x86_64-linux-gnu.so 
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fffff589000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ffffffc4000)

adding "cffi_libraries": ["m"] to the passed options works, as

root@7f816068061f:/home/tests# ldd /root/.cache/fenics/libffcx_forms_1983535cfbcc858b2908aa5664d28236c046a33f.cpython-310-x86_64-linux-gnu.so 
	libmvec.so.1 => /lib/x86_64-linux-gnu/libmvec.so.1 (0x00007fffff6b5000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fffff48c000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fffff3a5000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ffffffc4000)

My suggestion is we always link against libm and do not rely on the python linkage.

[garth-wells]

Fixed in #670.