Can't build on an M1/ARM Mac

[cpaika]

What happens?

I'm attempting to clone this repo to make changes. I cloned the repo, and initialized the duckdb submodule. When I run make it fails 82% of the way in building the parquet duckdb extension:

[ 82%] Built target tpch_loadable_extension
[ 82%] Linking CXX shared library parquet.duckdb_extension
Undefined symbols for architecture arm64:
  "_parquet_init", referenced from:
     -exported_symbol[s_list] command line option
  "_parquet_version", referenced from:
     -exported_symbol[s_list] command line option
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [extension/parquet/parquet.duckdb_extension] Error 1
make[2]: *** [extension/parquet/CMakeFiles/parquet_loadable_extension.dir/all] Error 2
make[1]: *** [all] Error 2
make: *** [release] Error 2

The same error occurs when I run GEN=ninja make as well.

To Reproduce

Clone this repository, initialize the submodule, and run make

OS:

Mac OS X

SQLite Version:

Default

DuckDB Version:

main

DuckDB Client:

N/A

Full Name:

Christopher Paika

Affiliation:

Tesla

Have you tried this on the latest main branch?

• [X] I agree

Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

• [X] I agree

[Mytherin]

Thanks for the report! I'm running on an M1 mac as well and not running into this problem - are you perhaps on an older MacOS/compiler version or using a custom/different compiler?

$> clang++ --version
Apple clang version 15.0.0 (clang-1500.1.0.2.5)
Target: arm64-apple-darwin23.2.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Alternatively do you perhaps have any compilation flags defined in your env that could change symbol visibility/linking behavior?

[cpaika]

Upgraded to Sonoma to match your clang++ version, still getting the exception unfortunately:

Apple clang version 15.0.0 (clang-1500.1.0.2.5)
Target: arm64-apple-darwin23.2.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

My InstalledDir is different, unsure if that matters if the versions are the same.

It seems like the problem is building the duckdb submodule, is there anything special I have to do to that other than check it out?

[Mytherin]

Hm, odd. I still suspect there's something different about the configuration/environment setup that causes the change in behavior. As a work-around you could try uncommenting this line which is likely where the problem originates from.

[breckcs]

I work for the same organization as @cpaika, so I have a similar M1 Mac setup. I can build duckdb (build/release/duckdb) and the sqlite_scanner extension (build/release/extension/sqlite_scanner/sqlite_scanner.duckdb_extension). My build fails later linking the parquet.duckdb_extension:

[ 91%] Linking CXX shared library parquet.duckdb_extension
ld: warning: ignoring duplicate libraries: '../../src/libduckdb_static.a', '../tpch/libtpch_extension.a', 'libparquet_extension.a'
ld: Undefined symbols:
  _parquet_init, referenced from:
      <initial-undefines>
  _parquet_version, referenced from:
      <initial-undefines>
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[3]: *** [extension/parquet/CMakeFiles/parquet_loadable_extension.dir/build.make:721: extension/parquet/parquet.duckdb_extension] Error 1
gmake[2]: *** [CMakeFiles/Makefile2:7770: extension/parquet/CMakeFiles/parquet_loadable_extension.dir/all] Error 2
gmake[1]: *** [Makefile:136: all] Error 2
gmake[1]: Leaving directory '/Users/cbreck/github/sqlite_scanner/build/release'
make: *** [release] Error 2

The steps I used:

1. git clone https://github.com/duckdb/sqlite_scanner.git
2. cd sqlite_scanner
3. git submodule init
4. git submodule update
5. make

My clang++ version:

% clang++ --version
Apple clang version 15.0.0 (clang-1500.1.0.2.5)
Target: arm64-apple-darwin23.2.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

@cpaika, the difference in InstalledDir may be because you don't have Xcode installed.

[carlopi]

I think this might come from the fact that somehow the linker links against a [lib]parquet_extension.so or some similar problem connected to dynamic libraries vs static libraries.

Can you try check something what the command ls build/relese/**/*.so returns? Or add some more logging to the build?

[breckcs]

If I comment out this line, it builds.

That line was originally introduced by @jkub here, then modified by @Mytherin here then here.

With CMAKE_VERBOSE_MAKEFILE set to ON, the output when the build works:

[ 91%] Linking CXX shared library parquet.duckdb_extension
cd /Users/cbreck/github/sqlite_scanner/build/release/extension/parquet && /opt/homebrew/Cellar/cmake/3.28.1/bin/cmake -E cmake_link_script CMakeFiles/parquet_loadable_extension.dir/link.txt --verbose=1
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -O3 -DNDEBUG -O3 -DNDEBUG   -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk -dynamiclib -Wl,-headerpad_max_install_names -o parquet.duckdb_extension -install_name @rpath/parquet.duckdb_extension CMakeFiles/parquet_loadable_extension.dir/column_reader.cpp.o CMakeFiles/parquet_loadable_extension.dir/column_writer.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_crypto.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_extension.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_metadata.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_reader.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_statistics.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_timestamp.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_writer.cpp.o CMakeFiles/parquet_loadable_extension.dir/serialize_parquet.cpp.o CMakeFiles/parquet_loadable_extension.dir/zstd_file_system.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/parquet/parquet_constants.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/parquet/parquet_types.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/thrift/thrift/protocol/TProtocol.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/thrift/thrift/transport/TTransportException.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/thrift/thrift/transport/TBufferTransports.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/snappy/snappy.cc.o "CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/snappy/snappy-sinksource.cc.o" CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/zstd_ddict.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/huf_decompress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/zstd_decompress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/zstd_decompress_block.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/entropy_common.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/fse_decompress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/zstd_common.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/error_private.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/xxhash.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/fse_compress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/hist.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/huf_compress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress_literals.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress_sequences.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress_superblock.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_double_fast.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_fast.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_lazy.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_ldm.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_opt.cpp.o  ../../src/libduckdb_static.a -Wl,-dead_strip ../tpch/libtpch_extension.a libparquet_extension.a ../../src/libduckdb_static.a ../tpch/libtpch_extension.a libparquet_extension.a ../../third_party/fsst/libduckdb_fsst.a ../../third_party/fmt/libduckdb_fmt.a ../../third_party/libpg_query/libduckdb_pg_query.a ../../third_party/re2/libduckdb_re2.a ../../third_party/miniz/libduckdb_miniz.a ../../third_party/utf8proc/libduckdb_utf8proc.a ../../third_party/hyperloglog/libduckdb_hyperloglog.a ../../third_party/fastpforlib/libduckdb_fastpforlib.a ../../third_party/skiplist/libduckdb_skiplistlib.a ../../third_party/mbedtls/libduckdb_mbedtls.a
ld: warning: ignoring duplicate libraries: '../../src/libduckdb_static.a', '../tpch/libtpch_extension.a', 'libparquet_extension.a'
gmake[3]: Leaving directory '/Users/cbreck/github/sqlite_scanner/build/release'

And when it fails:

[ 91%] Linking CXX shared library parquet.duckdb_extension
cd /Users/cbreck/github/sqlite_scanner/build/release/extension/parquet && /opt/homebrew/Cellar/cmake/3.28.1/bin/cmake -E cmake_link_script CMakeFiles/parquet_loadable_extension.dir/link.txt --verbose=1
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -O3 -DNDEBUG -O3 -DNDEBUG   -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk -dynamiclib -Wl,-headerpad_max_install_names -o parquet.duckdb_extension -install_name @rpath/parquet.duckdb_extension CMakeFiles/parquet_loadable_extension.dir/column_reader.cpp.o CMakeFiles/parquet_loadable_extension.dir/column_writer.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_crypto.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_extension.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_metadata.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_reader.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_statistics.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_timestamp.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_writer.cpp.o CMakeFiles/parquet_loadable_extension.dir/serialize_parquet.cpp.o CMakeFiles/parquet_loadable_extension.dir/zstd_file_system.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/parquet/parquet_constants.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/parquet/parquet_types.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/thrift/thrift/protocol/TProtocol.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/thrift/thrift/transport/TTransportException.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/thrift/thrift/transport/TBufferTransports.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/snappy/snappy.cc.o "CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/snappy/snappy-sinksource.cc.o" CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/zstd_ddict.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/huf_decompress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/zstd_decompress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/zstd_decompress_block.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/entropy_common.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/fse_decompress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/zstd_common.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/error_private.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/xxhash.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/fse_compress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/hist.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/huf_compress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress_literals.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress_sequences.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress_superblock.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_double_fast.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_fast.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_lazy.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_ldm.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_opt.cpp.o  ../../src/libduckdb_static.a -Wl,-dead_strip -Wl,-exported_symbol,_parquet_init -Wl,-exported_symbol,_parquet_version -Wl,-exported_symbol,_parquet_storage_ini* ../tpch/libtpch_extension.a libparquet_extension.a ../../src/libduckdb_static.a ../tpch/libtpch_extension.a libparquet_extension.a ../../third_party/fsst/libduckdb_fsst.a ../../third_party/fmt/libduckdb_fmt.a ../../third_party/libpg_query/libduckdb_pg_query.a ../../third_party/re2/libduckdb_re2.a ../../third_party/miniz/libduckdb_miniz.a ../../third_party/utf8proc/libduckdb_utf8proc.a ../../third_party/hyperloglog/libduckdb_hyperloglog.a ../../third_party/fastpforlib/libduckdb_fastpforlib.a ../../third_party/skiplist/libduckdb_skiplistlib.a ../../third_party/mbedtls/libduckdb_mbedtls.a
ld: warning: ignoring duplicate libraries: '../../src/libduckdb_static.a', '../tpch/libtpch_extension.a', 'libparquet_extension.a'
ld: Undefined symbols:
  _parquet_init, referenced from:
      <initial-undefines>
  _parquet_version, referenced from:
      <initial-undefines>
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[3]: *** [extension/parquet/CMakeFiles/parquet_loadable_extension.dir/build.make:724: extension/parquet/parquet.duckdb_extension] Error 1
gmake[3]: Leaving directory '/Users/cbreck/github/sqlite_scanner/build/release'
gmake[2]: *** [CMakeFiles/Makefile2:7773: extension/parquet/CMakeFiles/parquet_loadable_extension.dir/all] Error 2
gmake[2]: Leaving directory '/Users/cbreck/github/sqlite_scanner/build/release'
gmake[1]: *** [Makefile:139: all] Error 2
gmake[1]: Leaving directory '/Users/cbreck/github/sqlite_scanner/build/release'
make: *** [release] Error 2