[Discussion] how to model GLIBC version for conan packages

[SSE4]

To help us debug your issue please explain:

• [ ] I've read the CONTRIBUTING guide.
• [ ] I've specified the Conan version, operating system version and any tool that can be relevant.
• [ ] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

/cc @rdeterre @bc-lee @elizagamedev

this is more generic version of https://github.com/conan-io/conan/issues/3963 related issues: https://github.com/bincrafters/community/issues/296 https://github.com/bincrafters/community/issues/267

historically, conan didn't model glibc version, relying only on compiler version for binary compatibility checks. this causes some issues in the past, as binaries compiled with the same compiler weren't compatible across various linux distributions. for instance, if we compiled libraries or executables via GCC 4.8 on Ubuntu 12.04, these executables and libraries fail to run/link on CentOS 7 machines. same situation with other popular Linux distributions, like RHEL, Fedora, etc.

[danimtb]

We should discuss this issue once we face the cross-building model one as this might affect the current settings and have further implications in Linux packages.

[scddev]

I am facing a similar issue regarding libc when building packages for glibc and musl (e.g. Alpine). Both are Linux, x86_64 but linked against a different libc.

We modified settings.yml like this:

compiler:
  ...
  gcc:
    ...
    libc: [None, glibc, musl]
  clang:
    ..
    libc: [None, glibc, musl]

This does not cover the fact of different versions of glibc or musl but covers at least our use case.

[vector-of-bool]

To note: There is more than just the glibc version that defines the system ABI on Linux. Things like dynamic linking SONAMEs to system libraries present another dimension of whether a given ELF can be loaded correctly on a given system. Issues with differing system libraries were my trouble with Qt, even if there were compatible glibc versions.

I have a local fork of Conan at $job and I experimented with the introduction of two settings: os_family and os_distribution. The former is more general, with values like RedHat, Debian, SUSE and Windows, while the latter is more specific, with things like CentOS, RHEL, Ubuntu, and OpenSUSE. Another setting, os_release would cover the major revisions of a particular OS family/distribution, with values like 7 (for RHEL 7 and CentOS 7), 16.04, or 19.04.

A tuple of (os_family, os_release) is almost always enough to distinguish the basic ABI of the system, since the libc version and important system SONAMEs are stable within a given release of a distribution. In my tests, I was able to successfully generate Qt packages using this scheme that reliably rebuilt between a Ubuntu and CentOS system without them accidentally sharing binaries.

Of course, not all packages are so sensitive to these system attributes, and may want to "opt-in" to become more granular.

One case this design does not consider is that a CentOS 6 package is probably compatible with CentOS 7 (as long as the proper SONAMEs match), but I personally don't consider that a compelling enough feature to warrant additional complexity.

[rdeterre]

As a workaround for this issue, is there a conan install option we could use to prevent a machine from using prebuilt binaries from the remote server, and rebuild packages from source if it does not have a locally built package?

Simply using --build solves the issue, except that all the dependencies get rebuilt every time even when perfectly valid prebuilt binaries are already available locally. This is problematic for big dependencies like Boost.

[Minimonium]

I mean, always building the stuff from public servers maybe not that nice, since then why do we store it all then. Indeed, it would be great to have a distribution + its version combination to define the whole distribution environment, not just glibc. But also we could consider providing an alternative approach of the glibc + standard library and its version combination for "compatible" binaries if your package doesn't use other system libraries. Maybe not that useful for public binaries, but it's good to have such default settings out of the box for private workshops to not reinvent the wheel.

[kwinsch]

Taking the suggestions from https://docs.conan.io/en/latest/extending/custom_settings.html as a base and incorporating the ideas from @scddev , the following custom settings are working for me:

os:
...
  Linux:
    distro: [None, RHEL6, RHEL7, Ubuntu18.04]
...
compiler:
  ...
  gcc:
    ...
    libc: [None, glibc, musl]
...

Whe also have some custom made distros for embedded device which we all have a distro name for. Unlike the glibc mess, working with musl as a libc, does usually not require the definition of the distro.

[kwinsch]

For the toolchain side, we also had to expand 'os_build', so that the from source generated toolchain works on all build nodes. The toolchain is added as a dependency in the profile via a '[build_requires]'. The following expansion of the 'settings.yml" seems to works for our embedded, cross compilation setup.

os_build: 
    Windows:
    WindowsStore:
    Linux:
        distro: [None, "alpine", "ubuntu1204", "ubuntu1804"]
    Macos:
    FreeBSD:
    SunOS:
    AIX:
...
os:
    ...
    Linux:
        distro: [None, "buildroot_foo", "buildroot_bar"]
        platform: [None, "foo", "bar"]
...
compiler:
  ...
  gcc:
    ...
    libc: [None, glibc, musl]
...

[datalogics-robb]

I'm in favor of the solution using libc and its version. Distribution is more specific, but is orthogonal to the things that make a binary incompatible. If a package is created linking to system shared libraries, those libraries might not have been installed even though the same version of the same distribution is used. This same issue could also be resolved by using static "linking", which isn't feasible to do with glibc.

[puetzk]

We (with several linux targets, some public distros and some yocto-based cross-toolchains) do the following in settings.yml

os:
    Linux:
        distribution:
            None:
            # based on /etc/os-release $ID and $VERSION_ID
            ubuntu:
                version: [ "16.04", "16.10", "18.04", "18.10" ]
            wrlinux:
                version: [ "7.0", "8.0", "8.0.0.28", "8.0.0.30" ]
            fsl-imx:
                version: [ "4.14.98-2.0.0_ga" ]

Allowing distribution: None covers the existing scheme if the profile doesn't set it, but otherwise we use the https://www.freedesktop.org/software/systemd/man/os-release.html standard to get labeling (not for the first time making me wish there was a .py version of profiles like there is conanfile.txt and conanfile.py, so it could just read os-release directly).

Yocto supports this sort of labeling via the bitbake recipe https://git.yoctoproject.org/cgit.cgi/poky/plain/meta/recipes-core/os-release/

[madebr]

What about this:

libc:
  glibc: ["2.18", "2.19", "2.20", ..., "2.31"]
  musl: ["0.9", "1.0", "1.1", "1.2"]
compiler:
  gcc:
    libc: &libc (None for undefined or default)
  llvm:
    libc: &libc
  Visual Studio:
    [no libc]

For auto generation of the profile, the libc version can be detected by compiling+executing a simple file using gnu_get_libc_version, or extracting the version defintions from objdump -p /lib64/libc.so.*.

The bigger problem is: how will the hashing/package id work?

Packages compiled with different glibc versions, will generate a different package id. But how should conan behave if a consumer has a newer glibc library? Should it require a package of the same glibc version? Should it look for all available packages having a glic version lower as its own? ALso, I"m afraid this range matching is incompatible with the hashing principle of conan.

[Minimonium]

That's the same issue as with compiler.cppstd. We need a way of constraining compatible settings from upstream.

[mpusz]

Another point to highlight the importance of this feature is https://github.com/conan-io/conan-docker-tools/issues/200. Even though there is gcc-9.3 available in Ubuntu 20.04 (used already for gcc-10 image) and it fixes some bugs in gcc-9.2 that prevent my library to compile on the older compiler, we can't upgrade the docker image because of glibc issue.

[prince-chrismc]

👋 Would it be possible to get an update on this issue please? It's tagged for Conan 2.0 and I haven't seen anything in the release notes but maybe it's sneaked past

[samuel-emrys]

What about this:

libc:
  glibc: ["2.18", "2.19", "2.20", ..., "2.31"]
  musl: ["0.9", "1.0", "1.1", "1.2"]
compiler:
  gcc:
    libc: &libc (None for undefined or default)
  llvm:
    libc: &libc
  Visual Studio:
    [no libc]

For auto generation of the profile, the libc version can be detected by compiling+executing a simple file using gnu_get_libc_version, or extracting the version defintions from objdump -p /lib64/libc.so.*.

The bigger problem is: how will the hashing/package id work?

Packages compiled with different glibc versions, will generate a different package id. But how should conan behave if a consumer has a newer glibc library? Should it require a package of the same glibc version? Should it look for all available packages having a glic version lower as its own? ALso, I"m afraid this range matching is incompatible with the hashing principle of conan.

@madebr Is this binary compatibility specification still a concern with the newer introduction of the compatibility() method, and compatibility.py in conan 2.0? Presumably these would allow the identification of any package with a libc version lower than the one being built is compatible (from a libc perspective).

I've been thinking about ways to better model libc with conan so that packages built on our CI will be binary compatible with all of our targets, and what you've specified here would be ideal.

[prince-chrismc]

Since I am working on the Conan 2.0 what's new binary compatibility slides for meeting c++ let me try to answer my own question lol - things have changed 😅

---

There's two problems highlighted in the thread:

• how do you model libc in settings.yml
• how do we express forward compatibility (thats generally assumed)

Currently with 1.x they needed to be answered together because the package ID needed the setting and the "great then minor" notion we use in daily lives.

The way compatibility.py works is you get to fallback to other package ID -- this will allow us to track the exact version of glibc in each binary and pick higher ones if they are not available.

I haven't the time to play with it yet but... I'd love to see a working demo 😍 DM me and we can do a zoom 😉

The settings.yml you quotes above + a compatibility implementation - has a really good chance at solving this.

[prince-chrismc]

That's the same issue as with compiler.cppstd. We need a way of constraining compatible settings from upstream.

This will be available by default in 2.0 :)

[memsharded]

This will be available by default in 2.0 :)

Yes, let me comment on this:

• The default compatibility.py in 2.0 will contain a "compatibility" definition for the different cppstd values, because we have learned that in general they are massively compatible, and we have the validation of ConanCenter, that has been merging different binaries built with different cppstd, and things work fine. We have only known about some exceptional case that they were incompatible.
• Unfortunately we cannot say the same about glibc, in which the compatibility with different versions is not that clear, and unless you just use a old version, and strictly guarantee only backwards, it is very possible to get problems with that.

For these reasons, and also because some performance limitations in the server API, that will take a few months after 2.0 to be resolved, having to validate multiple values of a potential "compatible" binary will take a lot of time, the glibc compatibility will not be the default in Conan 2.0. We hope that with the compatibility.py plugin, users wanting to manage this, will be able to easily achieve what they want.

[samuel-emrys]

The settings.yml you quotes above + a compatibility implementation - has a really good chance at solving this.

This is great news.

Unfortunately we cannot say the same about glibc, in which the compatibility with different versions is not that clear, and unless you just use a old version, and strictly guarantee only backwards, it is very possible to get problems with that.

I was under the impression that glibc provided perfect backward compatibility, but it looks like your intuition here is correct. I did a bit of research, and I found a method of checking ABI compliance on the glibc wiki. The results are also published here, which demonstrate that from one version to another, there's generally >95% backwards compatibility version-on-version. This actually looks like quite a useful site, because it details exactly which symbols have been added/removed that affect backward ABI compatibility.

From a conan perspective, this might be enough to use perfect backward compatibility as a generally-correct heuristic with an ability to override if it actually presents an issue. An argument could equally be made the other way though, if the overall philosophy to be applied is conservatism.

Beyond defining what is and isn't compatible though, I think it is important to capture what the binary compatibility is - so the libc implementation and version, even if binary compatibility can't be easily inferred automatically. This would at least allow one to reason about the binary compatibility of any given binary.

[memsharded]

Thanks @samuel-emrys

That is very useful information, I think what you are proposing (perfect backward compatibility) could be a candidate to be added as default later in time. As I commented, the performance issue will take some time to fix, so that will give time for users to try their custom compatibility.py and if the feedback in practice keeps consistent, we could consider making it the default built-in one.

Beyond defining what is and isn't compatible though, I think it is important to capture what the binary compatibility is - so the libc implementation and version, even if binary compatibility can't be easily inferred automatically.

The problem is that we have no mechanism for this, and it would be challenging to do it. Whatever is captured in the conaninfo.txt, becomes part of the package_id by construction, so it would require a new feature, seems too much at this moment, specially if we think that there is some possibility that it could be made default later down the road.

[datalogics-robb]

This is a perfect solution! In practice, I have never seen a case where glibc was not upward-compatible – across different distros C programs and libraries need only to run on a system with a glibc at the same or higher version (without any linking tricks).

[prince-chrismc]

Thanks for clarifying my comment, the extra two sentences are important!

---

I've professionally encountered problems with GLIBC compatibility. We had a customer running a Linux from scratch whose exact version had a bug that only triggered with our service with libdispatch trying to fork a child process. It was only one exact version that had the bug, the rest...

There are problems.. 95% sounds right

It all depends on you line of work - some have more 💩 then others 🤣

[mpusz]

The default compatibility.py in 2.0 will contain a "compatibility" definition for the different cppstd values, because we have learned that in general they are massively compatible, and we have the validation of ConanCenter, that has been merging different binaries built with different cppstd, and things work fine. We have only known about some exceptional case that they were incompatible.

@memsharded Well, I am not sure if that is correct. There are two factors here:

1. The binary compatibility of the C++ Standard ABI and C++ Standard Library
2. Library's code depending on the C++ version flag.

As long as conan-io/conan#1 is not a major problem, as you stated above, conan-io/conan#2 is a major issue. There are a lot of projects and libraries that query the C++ version preprocessor macro and provide a different ABI based on it. One of the most common examples is a usage of a custom or STD variant, optional, and string_view in the code base depending on the C++17 availability. For example: https://github.com/martinmoene/variant-lite/blob/master/include/nonstd/variant.hpp.

I already supported two customers that had above mentioned ABI issues as they compiled the project with C++17 while a dependency was using C++14, and both depended on the same variant-lite package.

[memsharded]

As long as https://github.com/conan-io/conan/issues/1 is not a major problem, as you stated above, https://github.com/conan-io/conan/pull/2 is a major issue. There are a lot of projects and libraries that query the C++ version preprocessor macro and provide a different ABI based on it. One of the most common examples is a usage of a custom or STD variant, optional, and string_view in the code base depending on the C++17 availability. For example: https://github.com/martinmoene/variant-lite/blob/master/include/nonstd/variant.hpp.

I guess that a way to inhibit the default cppstd compatibility from recipes would be useful.

[datalogics-robb]

I've professionally encountered problems with GLIBC compatibility. We had a customer running a Linux from scratch whose exact version had a bug that only triggered with our service with libdispatch trying to fork a child process. It was only one exact version that had the bug, the rest...

IMHO - we're discussing a "model" for glibc compatibility for Linux - bugs are not the sort of thing I would include in a model of how glibc compatibility works. Bugs will invariably violate any rules for compatibility, and render any model invalid. I would recommend creating a model that ignores a special case like a bug in a particular version of a library as it will cause more problems than it will solve. A more reasonable fix for an issue like this might be to have a recipe require a version of glibc later than the problematic one.

[blackliner]

We are also facing the issue of Ubuntu 20.04 binaries of dependencies being pulled for 18.04 builds (because package IDs are the same on Ubuntu 18.04 and 20.04), and erroring out with:

~/.conan/data/opencv/4.5.2/_/_/package/b2b0041ab1b097316770fce0d8e5c4930aed6469/lib/libopencv_imgcodecs.so: undefined reference to `pow@GLIBC_2.29'
~/.conan/data/opencv/4.5.2/_/_/package/b2b0041ab1b097316770fce0d8e5c4930aed6469/lib/libopencv_imgproc.so: undefined reference to `exp@GLIBC_2.29'
~/.conan/data/opencv/4.5.2/_/_/package/b2b0041ab1b097316770fce0d8e5c4930aed6469/lib/libopencv_imgproc.so: undefined reference to `log@GLIBC_2.29

Do we have a workaround that works today?

[samuel-emrys]

@blackliner The solution that I most often see recommended is to implement the following in your own settings.yml:

libc:
  glibc: ["2.18", "2.19", "2.20", ..., "2.31"]
  musl: ["0.9", "1.0", "1.1", "1.2"]
compiler:
  gcc:
    libc: &libc (None for undefined or default)
  llvm:
    libc: &libc
  Visual Studio:
    [no libc]

With conan 2.0, you could also try to define the compatibility relationship between different versions of glibc as well, using compatibility.py

[maksim-petukhou]

It's absolutely fine to use binaries compiled for centos 7 (glibc 2.17) in ubuntu 16.04 (glibc 2.23), because glibc is backwards compatible. So I wouldn't go that way:

os:
    Linux:
        distribution:
            None:
            # based on /etc/os-release $ID and $VERSION_ID
            ubuntu:
                version: [ "16.04", "16.10", "18.04", "18.10" ]
            wrlinux:
                version: [ "7.0", "8.0", "8.0.0.28", "8.0.0.30" ]
            fsl-imx:
                version: [ "4.14.98-2.0.0_ga" ]

This solution is not a valid yaml and it exposes libc key:

libc:
  glibc: ["2.18", "2.19", "2.20", ..., "2.31"]
  musl: ["0.9", "1.0", "1.1", "1.2"]
compiler:
  gcc:
    libc: &libc (None for undefined or default)
  llvm:
    libc: &libc
  Visual Studio:
    [no libc]

I modified it like so:

compiler:
    gcc:
        ...
        libc: &libc
            None: ~
            glibc: 
                version: ["2.17", ...]
    msvc:
        ...
        # no libc
    clang:
        ...
        libc:
            <<: *libc

None is for compatibility (null in Conan 2.0). There is a little problem with packages for tools like ninja, tar etc. They have this in their conanfile:

    def package_id(self):
        del self.info.settings.compiler

It is a problem because an executable built for glibc 2.35 won't start with glibc 2.17. In order to fix this we should del everything in settings.compiler except libc if it is present.

[jusito]

because glibc is backwards compatible.

Are you sure with it because as far as I know its not: https://abi-laboratory.pro/?view=timeline&l=glibc I don't have deep insights into this topic, just read about it and found this overview some time ago.

[maksim-petukhou]

because glibc is backwards compatible.

Are you sure with it because as far as I know its not: https://abi-laboratory.pro/?view=timeline&l=glibc I don't have deep insights into this topic, just read about it and found this overview some time ago.

@jusito let me just quote official webpage:

The GNU C Library is designed to be a backwards compatible, portable, and ...

There you can find some details on how they handle it

[datalogics-robb]

And to add to that – I’ve been building proprietary “shrink wrap” software on Linux for 20+ years, and I have yet to see an instance where C/C++ software compiled on Linux glibc that had trouble running on a system because it had a newer glibc (or maybe my memory is failing). 😉

IMHO I think it would be a surprise if the glibc version were part of the compiler settings though, as it’s a property of the OS/distro.

From: Maksim Petukhov @.> Date: Tuesday, March 7, 2023 at 8:56 AM To: conan-io/conan @.> Cc: Rob Boehne @.>, Comment @.> Subject: Re: [conan-io/conan] [Discussion] how to model GLIBC version for conan packages (#3972)

because glibc is backwards compatible.

Are you sure with it because as far as I know its not: https://abi-laboratory.pro/?view=timeline&l=glibc I don't have deep insights into this topic, just read about it and found this overview some time ago.

@jusitohttps://github.com/jusito let me just quote official webpagehttps://www.gnu.org/software/libc/:

The GNU C Library is designed to be a backwards compatible, portable, and ...

Therehttps://developers.redhat.com/blog/2019/08/01/how-the-gnu-c-library-handles-backward-compatibility you can find some details on how they handle it

— Reply to this email directly, view it on GitHubhttps://github.com/conan-io/docs/issues/3960, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACVSF3QWMA4J64BZRUU4ABLW25EAZANCNFSM4GFTLHYQ. You are receiving this because you commented.Message ID: @.***>