amazonlinux
[Package Update Request] - OpenSSL 3.2.x
What package is missing from Amazon Linux 2023? Please describe and include package name.
openssl 3.2+ (or openssl 3.1+)
Is this an update to existing package or new package request?
This is an update to the existing openssl 3.0.8.
Is this package available in Amazon Linux 2? If it is available via external sources such as EPEL, please specify.
No
Any additional information you'd like to include. (use-cases, etc)
You might want to merge this in as a duplicate of https://github.com/amazonlinux/amazon-linux-2023/issues/512 , but I wanted to raise this for 3.1+ in particular, for a particular use-case: a significant performance regression compared to Amazon Linux 2.
We recently upgraded some servers from AL2 to AL2023 so that we could upgrade to PHP 8.2 on elastic beanstalk. During testing we found that some of our servers which perform lots of network requests to other servers over an HTTPS REST API were performing way worse - something of the order of a four or five times or more increase in CPU load, it seems. We had to scale some t3.medium servers with lots of headroom up to t3.2xlarge, and others even further - which can add up!
I managed to mostly reproduce this on base AL2 and AL2023 servers, and using a simple curl command: curl -w "tcp_established: %{time_connect}, ssl_handshake_done: %{time_appconnect}, TTFB: %{time_starttransfer}\n" -o /dev/null -s --head "https://aws.amazon.com/". On a t2.micro, AL2 shows the SSL stage taking ~9-10ms, and on AL2023 it takes ~37-40ms. This comes with an accompanying increase in CPU usage; I assume that other SSL tasks also vary, as our PHP servers seem to show a larger hit than this (and initial SSL negotiation would only be a tiny part of their tasks!).
Long story short, I tried various things like disabling SELinux and ensuring TLS1.2, but in the course of a still-in-progress support case (170690787201370) a helpful Linux Team member pointed out the OpenSSL version differences. This led me to the epic https://github.com/openssl/openssl/issues/17064 which is still open but does show that some work has happened in OpenSSL 3.1 and 3.2 to address some of the regressions.
I tried building OpenSSL from source on a clean AL2023 box and ended up with these numbers:
- AL2 (out-of-the-box OpenSSL 1.0.2k-fips): ~9-10ms SSL negotiation
- AL2023 (out-of-the-box OpenSSL 3.0.8): ~37-40ms SSL negotiation
- AL2023 (OpenSSL 3.0.8, build with default
make): ~38-42ms SSL negotiation - AL2023 (OpenSSL 3.2.1, build with default
make): ~12-13ms SSL negotiation
So still an overhead compared to AL2, but OpenSSL 3.2 improves the performance regression significantly.
(I'm sort of surprised I haven't seen more about this when trying to work out why our AL2023 instances were performing so much worse!)
I saw the comment on https://github.com/amazonlinux/amazon-linux-2023/issues/512#issuecomment-1757791113 indicating that there are likely complex reasons openssl generally doesn't get updated - but maybe an alternate version package could be offered, like some of the older versions left available on AL2? (I see FIPS may be one of those reasons but the openssl source page mentions 3.2 can be built with the 3.0.8 FIPS provider...)
Wonder if this is related at all to https://github.com/amazonlinux/amazon-linux-2023/issues/434 🤔
There's FIPS implications for bumping OpenSSL, as well as various compatibility things to ensure we address. It does look like OpenSSL is making this kind of thing more feasible, but it's still a big step for us to evaluate the possibility of any update.
Just setting expectations that this is less likely than other package update requests we have.
Understood 😬 Is there a chance of offering more recent versions as an alternate package, or does that not really align with FIPS compliance, or be too much of a pain to maintain?
Ran into a similar issue when I bumped a Lambda function from Python 3.11 -> 3.12 (and therefore AL2/OpenSSL 1.x -> AL2023/OpenSSL 3.0.x). A simple Lambda that posted an SNS message took 2x longer with 3.12. When I build a custom layer with OpenSSL 3.2.x on Python 3.12 it brought it back in-line with performance on 3.11.
FIPS provider module (fips.so) can be used from the FIPS-certified 3.0.x version, in a 3.1/3.2/3.3 series.
This enables e.g. OpenSSL 3.3.1 to support FIPS by loading fips.so compiled from 3.0.x.
Is there any news on this from AWS?
The performance degradation seen in lambda appears to be resolved as they are now using openssl 3.1.5 in both the AL2 and AL2023 runners.
Is it possible to get a non-fips AL2023 AMI, or a fips-specific AL2023 AMI.
Considering the the Amazon Linux 2023 release notes say
Known issues - AL2023 is not yet FIPS certified. AL2023 is in the process of being certified for FIPS 140-3.
surely it would be ok to have non-FIPS certified images considering that is what they currently are anyway.
We are working on both backporting of some of the performance improvements and possibly updating to a new version, stay tuned.
As for the mention about certification, the existing version has been in the certification pipeline since 2023 so if we update it we lose that and go back to the starting block, not an option. However what @krk mentioned (using the certified FIPS module with the new version) is an option.
There are other issues with updating though. Some things that built with 3.0 don't with 3.2 in our experience, but we are looking into it.
Thanks for the update @ozbenh. Is there any specific place we should stay tuned for when there is more to report on it? Like just stay following this issue or is there any other thing we should subscribe to?
Unfortunately, Amazon Linux 2023 version 2023.6.20241209 didn't come with any backports to openssl as far as I can see.
Yes we chose not to take too much risks right before the xmas break and around other end-of-year events. You might have noticed out release cadence has slowed down. I'm expecting this to come out once things resume to normal
Is the latest AL2023 looking to support OpenSSL 3.1 - 3.2 soon? Is there a timeline for this?
Amazon Linux 2023.7 (2023.7.20250331) has rebased OpenSSL to v3.2.2. Please test.
See also, AL2023.7 Release Notes: https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.7.html
Nice, thanks - can see v3.2.2 is now built in, and tested the above curl handshake on 2023.7.20250331 and can confirm the SSL handshake is back down to ~11-14ms. As a reminder AL2 (out-of-the-box OpenSSL 1.0.2k-fips) was ~9-10ms but earlier AL2023 (out-of-the-box OpenSSL 3.0.8) was ~37-40ms!
It'll take a little while to test this with our full prod traffic in a different app setup but based on the above numbers this does look solved to me 👍🏻 Thank you!