ceph
ceph-dev: revive centos 8 builds for quincy and reef using vault.centos.org repos
point at vault.centos.org mirrors for centos 8 as suggested by @k0ste on the ceph-users mailing list thread Reef 18.2.4 EL8 packages ?
revert the commits from https://github.com/ceph/ceph-build/pull/2235 that removed centos 8 from the build matrix for quincy and reef branches
Thanks Casey! Can be this merged? Currently almost impossible to operate with nodes due dnf fatal errors, caused by 404
TASK [dnf : dnf | Install dnf-automatic package] ********************************************************************************************
fatal: [postgres4.example.com]: FAILED! => {"changed": false, "msg": "Failed to download metadata for repo 'ceph-reef': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", "rc": 1, "results": []}
Ah, so this is the cause of all of the failures we have this morning, with Ceph RPMs disappearing under our feet...
Dropping support of a major distribution should not be happening in the middle of a major version of any product, as this is a major change. And if it has to happen, older minor versions of such product should not disappear out of thin air!
@k0ste as far as i know, this is still pending discussion
Discussion, does it really take a weeks to fix something that's broken? As predicted, the community is facing not only package issues, but also container issues
P.S.: I didn't expect so many kudos 👍, hopefully this will help bring some common sense back into the release process 🙂↕️
Any news? I'm still stuck with a cluster running 18.2.4 for 3 daemons and 18.2.2 for the remaining ones, hoping that at a certain point the CentOS Stream 8 build will be resumed and the related images pushed to quay.io...
Any news? I'm still stuck with a cluster running 18.2.4 for 3 daemons and 18.2.2 for the remaining ones, hoping that at a certain point the CentOS Stream 8 build will be resumed and the related images pushed to quay.io...
Added tracker for @zdover23 help, about documentation of package drop feature
the infrastructure team has so far been unwilling to reinstall centos stream 8 on any of the lab machines to process these builds
the infrastructure team has so far been unwilling to reinstall centos stream 8 on any of the lab machines to process these builds
Well, that's disappointing. As many others already wrote, such a big jump in system requirements in a point release should never happen. I'll have to figure out how to build Ceph in a Rocky Linux 8 environment and then assemble an image for Cephadm.
and then assemble an image for Cephadm
@nicolamori can you please clarify what's wrong with the released 18.2.4 container image?
I can't speak for @nicolamori, but we have three clusters, total ~1800 osd, running Alma 8.10, with no containers, no cephadm, just plain rpm install. Converting everything to containers is a massive change - we would see less risk in upgrading the OS to Alma 9.4 and then carrying on using rpms. Though the work involved is if course far higher. Thanks, Jake
and then assemble an image for Cephadm
@nicolamori can you please clarify what's wrong with the released 18.2.4 container image?
@cbodley I run a Cephadm cluster with old hardware, some machines do not support x86-64-v2 and this caused the upgrade to 18.2.4 fail at half way, leaving me with three daemons running 18.2.4 and the others running 18.2.2. See the link posted by k0ste above for details.
and then assemble an image for Cephadm
@nicolamori can you please clarify what's wrong with the released 18.2.4 container image?
The another problem of container due distro change: https://github.com/rook/rook/issues/14502 (tracker !67213)
the infrastructure team has so far been unwilling to reinstall centos stream 8 on any of the lab machines to process these builds
This is understandable, it is an outdated and unsupported distribution. But why not use an active and supported distribution like Rocky Linux 8 or AlmaLinux 8? A lot of CEPH installations are running on them, as can be seen from the comments in this thread.
This is understandable, it is an outdated and unsupported distribution. But why not use an active and supported distribution like Rocky Linux 8 or AlmaLinux 8? A lot of CEPH installations are running on them, as can be seen from the comments in this thread.
The packages can be build on mock, for any supported distro, in container. I think, for this time - everyone who needs this packages: already builded it internally. Just another lesson
the infrastructure team has so far been unwilling to reinstall centos stream 8 on any of the lab machines to process these builds
This is understandable, it is an outdated and unsupported distribution. But why not use an active and supported distribution like Rocky Linux 8 or AlmaLinux 8? A lot of CEPH installations are running on them, as can be seen from the comments in this thread.
As others pointed out, it's not so understandable that support for one OS is drop in the middle of a release cycle, but whatever. I switched to Ubuntu 22.04 for my Cephadm image, I build it using the official Ceph packages. It does not require x86-64-v2, and nor it does Ubuntu 24.04, so I hope I will also be able to upgrade to Squid on my outdated hardware.
Any news? I'm still stuck with a cluster running 18.2.4 for 3 daemons and 18.2.2 for the remaining ones, hoping that at a certain point the CentOS Stream 8 build will be resumed and the related images pushed to quay.io...
Added tracker for @zdover23 help, about documentation of package drop feature
@k0ste: I'm ready to update the documentation, but it's not clear to me what I should say in that update. Let me know, and I'll do it immediately.
@k0ste: I'm ready to update the documentation, but it's not clear to me what I should say in that update. Let me know, and I'll do it immediately.
It is necessary to describe that support for building packages using Ceph CI may be discontinued in the middle of the release cycle. Due to the unwillingness of the infrastructure team to maintain a container/virtual machine for building packages for a specific branch of the distribution (or any other). Additional information can be obtained from the people who made this decision, since no consultation with the community was conducted. And the telemetry data that the community sends was apparently also not studied. Personally, I, like other community members, only received a result in the form of 404 🥹
It should also document that https://download.ceph.com/rpm-reef/ and the like is NOT a stable source to get RPMs from, since existing RPMs were removed from there. I repeat myself, but existing RPMs should not have been removed from there, period, regardless of dropping support for RHEL8 on new dot versions.
We have a 3 PB Cephfs running on 16 nodes with 200 OSD's. All 16 nodes are on Rocky 8 and are running 18.2.2. Could someone kindly point me to a "How To" on upgrading the underlying OS(Rocky8 to Rocky9) on a ceph system that is in production? Thank you in advance.
