Conan integration to import third-party C/C++ dependencies

[aherrmann]

This PR adds support for importing third-party C/C++ dependencies provided by the Conan package manager.

Detailed motivation and design are contained in the module doctring of the main toolchain module under prelude/toolchains/conan/defs.bzl. In brief, some C/C++ third-party dependencies are difficult to import into a Buck2 project, e.g. the current self-hosted build of Buck2 vendors a binary openssl and skips jemalloc for that reason.

The approach proposed here is to instead use the existing package build definitions for the Conan package manager and build these third-party dependencies with Conan. This approach is similar to how rules_haskell integrates with Stack and Cabal to import third-party Haskell dependencies, or how JVM packages are imported with the help of Coursier in Bazel.

Conan was chosen for its relatively large community package index, compatibility with Linux, MacOS, and Windows, configurability and extensibility, and its support for cross-compilation. Alternatively, vcpkg was considered. However, its choice of CMake as implementation and configuration language makes it harder to integrate. Other interesting comparisons can be found here and here.

This integration aims to leave Buck2 in charge of the overall build, when which targets are built, and which configurations and toolchains are used.

Code map:

• prelude/toolchains/conan/defs.bzl the main Starlark module for the Conan integration. Defines the relevant rules.
• prelude/toolchains/conan/*.py Python scripts implementing the various build actions.
• prelude/toolchains/conan/buckler/ a Conan extension that defines a generator to generate Buck2 import targets for Conan built packages.
• examples/prelude/toolchains/BUILD an example Conan toolchain configuration.
• examples/prelude/cpp/conan/ an example Conan use-case.

This PR adds tests of the Conan example to the Linux and MacOS CI pipelines. Windows support will be added separately.

Remaining work for future PRs

• [ ] Better cross-platform support. Use Conan lock-file bundles to define cross-platform import targets.
• [ ] Windows support in particular. Currently builds fail on long-path issues, Conan provides workarounds for these issues that need to be investigated.
• [ ] Import Buck2’s third-party C/C++ dependencies with Conan.
• [ ] Cross-compilation support. Conan has builtin cross-compilation support that needs to be made use of.
• [ ] Test some more complex packages. Builds of some more complex packages failed when attempted with a zig cc toolchain, e.g. libmysqlclient fails to find libresolv.so. Investigate these issues. An example setup with the Zig toolchain can be found here.

[arlyon]

Excellent, looking forward to reading through this :)

[arlyon]

It would be cool to consider this option as well: https://docs.conan.io/en/1.46/reference/conanfile/tools/system/package_manager.html#configuration-properties-that-affect-how-system-package-managers-are-invoked

[arlyon]

This is great. I would love to see the package component limitation lifted though it seems like simply a matter of implementing it.

https://github.com/facebookincubator/buck2/blob/76c18df1fe5a0ef8e611779eb080bdcc606b1595/prelude/toolchains/conan/defs.bzl#L344

Otherwise, the code is succinct and clear. The python scripts seem solid, and I've managed to run :update for boost and glm, glfw, and glad and the output file looks correct and includes transitive deps. It even told me I had a missing system package which is cool. It's great that components are included (and I love the syntax) and dependencies are generated though I couldn't test the transitive deps due to the aforementioned issue.

I will let Neil dispense buck2 related wisdom but I am excited that this is ready to land. Hope you have a lovely holiday.

[aherrmann]

[...] I've managed to run :update for boost and glm, glfw, and glad [...] though I couldn't test the transitive deps due to the aforementioned issue.

Thanks for testing!

This is great. I would love to see the package component limitation lifted though it seems like simply a matter of implementing it.

I looked into these on Conan central and tried to build boost. It looks like the issue is that boost contains some header-only components for which len(libs) == 0. I've added a case to handle this and update the PR. With that I was able to build boost locally.

For the multiple library components case, i.e. len(libs) > 1, I haven't found an example yet to test this on, so haven't attempted implementing it, yet. Are you aware of such a Conan package? I looked into ffmpeg, which seems to have been one of the packages motivating components, but even that seems to use single library components by now as far as I can tell.

[yeswalrus]

Just curious, has this PR been abandoned? This functionality would be tremendously useful for me.

[f0rmiga]

@aherrmann Interesting work here. Is there something I could do to help push it across the finish line?

[aherrmann]

Thank you for the positive feedback!

In terms of implementation work this PR is ready. However, the main branch has moved on a fair bit since then, so, it will need a rebase and likely a fair bit of work to adapt to any relevant changes to main since.

@ndmitchell IIRC last we spoke about this PR the intention was to get this merged. Apart from a rebase, what would it take to get this in?

[ndmitchell]

Let's rebase, and do the minimum required to merge it. Basically focus on the changes to the existing toolchain. If users want the new code, and it's off to the side so can't have any negative effects, we should err on the side of merging it.

[f0rmiga]

That's amazing. Thanks for the quick reply.

[aherrmann]

@ndmitchell I've rebased this on main and made the necessary fixes to get CI green again. I no longer seem to have push access to update the branch of this PR, so I've opened #440 instead. PTAL