[i386] Build system does not check whether AVX are actually supported by assembler

[barracuda156]

Tried to build aom on i386 (Core Duo) with GCC. Build misdetects AVX as supported and that breaks it:

[  7%] Building CXX object CMakeFiles/yuv.dir/third_party/libyuv/source/row_gcc.cc.o
/opt/local/bin/g++-mp-13  -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_multimedia_aom/aom/work/aom-3.8.1 -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_multimedia_aom/aom/work/build -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_multimedia_aom/aom/work/aom-3.8.1/apps -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_multimedia_aom/aom/work/aom-3.8.1/common -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_multimedia_aom/aom/work/aom-3.8.1/examples -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_multimedia_aom/aom/work/aom-3.8.1/stats -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_multimedia_aom/aom/work/aom-3.8.1/third_party/libyuv/include -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_multimedia_aom/aom/work/aom-3.8.1/third_party/libwebm -pipe -Os -DNDEBUG -isystem/opt/local/include/LegacySupport -I/opt/local/include -D_GLIBCXX_USE_CXX11_ABI=0  -std=c++11 -Wall -Wdisabled-optimization -Wextra -Wextra-semi -Wfloat-conversion -Wformat=2 -Wlogical-op -Wpointer-arith -Wshadow -Wsign-compare -Wtype-limits -Wuninitialized -Wunused -Wvla -Wc++14-extensions -Wc++17-extensions -Wc++20-extensions -Wstack-usage=240000 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -arch i386 -mmacosx-version-min=10.6 -fPIC -Wno-shadow -MD -MT CMakeFiles/yuv.dir/third_party/libyuv/source/row_gcc.cc.o -MF CMakeFiles/yuv.dir/third_party/libyuv/source/row_gcc.cc.o.d -o CMakeFiles/yuv.dir/third_party/libyuv/source/row_gcc.cc.o -c /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_multimedia_aom/aom/work/aom-3.8.1/third_party/libyuv/source/row_gcc.cc
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_multimedia_aom/aom/work/aom-3.8.1/third_party/libyuv/source/row_gcc.cc:557:no such instruction: `vbroadcastf128 __ZN6libyuvL23kShuffleMaskARGBToRGB24E-L7$pb(%eax),%ymm6'
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_multimedia_aom/aom/work/aom-3.8.1/third_party/libyuv/source/row_gcc.cc:558:no such instruction: `vmovdqa __ZN6libyuvL15kPermdRGB24_AVXE-L7$pb(%eax),%ymm7'
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_multimedia_aom/aom/work/aom-3.8.1/third_party/libyuv/source/row_gcc.cc:560:no such instruction: `vmovdqu (%edx),%ymm0'
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_multimedia_aom/aom/work/aom-3.8.1/third_party/libyuv/source/row_gcc.cc:561:no such instruction: `vmovdqu 0x20(%edx),%ymm1'
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_multimedia_aom/aom/work/aom-3.8.1/third_party/libyuv/source/row_gcc.cc:562:no such instruction: `vmovdqu 0x40(%edx),%ymm2'
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_multimedia_aom/aom/work/aom-3.8.1/third_party/libyuv/source/row_gcc.cc:563:no such instruction: `vmovdqu 0x60(%edx),%ymm3'
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_multimedia_aom/aom/work/aom-3.8.1/third_party/libyuv/source/row_gcc.cc:565:no such instruction: `vpshufb %ymm6,%ymm0,%ymm0'

Configure checks for flags support, but that is not sufficient; compiler may support flags, but assembler may not support instruction set.

[jedisct1]

Like your other reports. You're mixing a 18 years old as command from an ancient Xcode version with a recently compiled gcc. You should use a consistent toolchain, either by sticking to Xcode for everything, or at least by using gcc with the GNU binutils (and a version from the same era).

[jedisct1]

This is not a bug in aom.

[barracuda156]

It is a bug, because the configure check does not do what it is designed to do. It does not work correctly.

I have provided you an example of configure of libopus which detects a lack of AVX successfully (on the same system): https://github.com/jedisct1/libsodium/issues/1350#issuecomment-1924177677

There is no problem with toolchain. In fact using Apple Unix tools is the recommended method by GCC upstream for building modern GCC, for example, on 10.5+.

[barracuda156]

UPD. The issue is with libyuv code in fact, aom just uses it.

UPD2. No, not only libyuv, aom own code too, for example aom_dsp fails likewise.

[barracuda156]

Worse is that even if one builds something with some swapping of an external assembler with these instructions enabled, the resulting binary will be broken, since the CPU does not support those (this is CoreDuo, it does not have AVX).

[jedisct1]

Compiling different implementations is expected. All these tools detect CPU capabilities at runtime (see x86_simd_caps() for example).