Use CMake "superbuild" as a "poor" package manager

[lum1n0us]

Background

The project highly depends on some open-source projects. Binaryren, Wabt and Wasi-sdk are necessary tools for generating Wasm files. LLVM provides libraries for JIT and AOT running mode. LibUV is another implementation of WASI supporting. More details.

Using OS provided package managers(apt, brew) to download/install all dependencies would require variants scripts on all development platforms (ubuntu, macos, even windowns). That's why we suggest vscode+devcontainer strongly. But it is not enough to cover everyone's requirement.

Maintain effort is another pain point. Version information and find_XXX() about dependencies are everywhere. It always requires patient and luck to change all places.

Rough idea

CMake only

CMake superbuild uses both find_xxx(for searching) and ExternalProject+FetchContent(for downloading) to implement a "poor" package manager. It allows CMake to check all dependencies and download missing ones in configuration phase and building phase if using FetchContent. It across platforms for sure. The downside is when a developer has to stick with Makefiles on some specific development platforms.

Git submodule only

Git submodule is another solution for dependencies but can be triggered ($git submodule update --init) by CMake on configuration phase with execute_process. It depends on Git instead of CMake. The downside is it is friendly for source-level-dependency but not for binary-level-dependency. Plus git clone --depth 1 a repo is still slower than downloading a source code archive when the repo is huge.

Together

If a dependency needs to build from source code, use submodule to maintain it. Use add_subdirectory or ExternalProject or FetchContent to import it. If a dependency needs binary only. Use ExternalProject or FetchContent to download it. Use a find_package to import it.

Reference:

cmake superbuild and git submodules

[sgerbino]

There is a fairly flexible package manager for CMake called Hunter (https://github.com/cpp-pm/hunter). I've used it for quite a while. It would allow you to retrieve and build all dependencies consistently. I'm currently experimenting with WAMR to embed in a larger project that uses Hunter. I have a little example project that embeds WAMR with JIT enabled using Hunter you can check out (https://github.com/sgerbino/wasmrunner).

It can build dependencies through a VCS like Git and also through git submodules. It uses CMake's Find* under the hood.

I've looked into Hunterizing WAMR and there are a handful of dependencies that would also need to be Hunterized. It's a fair bit of work but it would clean up the entire build process considerably.

[sgerbino]

I've been looking at different solutions for C++ dependency handling for a new project. Honestly, it's been kind of a mess for years. However, there are some new features for CMake that may be starting to clean things up: dependency_provider.

I just managed to get an example of how to use this working I think it is an interesting approach, it allows you to use any package manager to satisfy find_package() calls in CMake. It seems that the most popular package managers are conan and vcpkg. Conan has a package for wasm-micro-runtime, and I've decided to use it.

This video was particularly relevant: https://www.youtube.com/watch?v=k76LN8dSxx4