.NET ARM Lambda function using .NET 7 RC1 fails to start when using a custom runtime

[martincostello]

Describe the bug

With .NET RC1 available today, I attempted to deploy a .NET Lambda function I have to use it via a custom runtime (deployment).

However, once deployed the tests for the Lambda function showed it was failing with an error. Digging into the logs in CloudWatch show that the function is failing to start.

Failed to load /var/task/libcoreclr.so, error: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /var/task/libcoreclr.so)

This appears to be due to the AWS Lambda runtime not having the correct version of GLIBC (2.27) for .NET 7 to run, likely due to a change in the native requirements for .NET itself.

I'm not currently aware of a way to get a .NET application to publish a specific version of GLIBC with the compiled application - if there is one, I'm all ears to unblock early adoption.

Expected Behavior

The Lambda function initializes successfully.

Current Behavior

The Lambda function fails to initialise with the following error:

Failed to load /var/task/libcoreclr.so, error: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /var/task/libcoreclr.so)
RequestId: {GUID} Error: Runtime exited with error: signal: segmentation fault
Runtime.ExitError

Reproduction Steps

An AWS Lambda function derived from martincostello/alexa-london-travel at commit bfc8db1850aa3b6463ffd23fc96b76a730222e61 when deployed to AWS Lambda using the provided.al2 runtime will fail to initialize.

Possible Solution

1. If there's a way to manually include the appropriate version with GLIBC as part of the dotnet publish output from the AWS Lambda global tool.
2. The AWS Linux 2 runtime has a newer version of GLIBC installed on it that is at least version 2.27.

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

Amazon.Lambda.RuntimeSupport 1.8.2

Targeted .NET Platform

.NET 7 Release Candidate 1

Operating System and version

Linux 4.14.255-276-224.499.amzn2.aarch64 #1 SMP Tue May 3 22:29:59 UTC 2022

[martincostello]

Also logged with Microsoft here: https://github.com/dotnet/runtime/issues/75622

[normj]

Hi @martincostello were you getting the error for ARM or X64? I have reproduced the problem for ARM but not for X64. I'm not sure what are options are yet because getting provided.al2 updated to have a newer version of GLIBC could be a tough sell to the Lambda team but I'll see what I can do.

[martincostello]

Sorry for not making that clear, this is with Graviton and arm64.

[martincostello]

I've switched the function to x64 and re-deployed it with .NET 7 and now it's working as expected.

For my use case here that's an acceptable workaround until we can come up with a solution for arm64.

[normj]

Thanks for confirming. I'll see what I can do on my side about it. I suspect we are stuck on the ARM side till we have a managed .NET 8 runtime or Lambda offers an AL2022 provided runtime.

[Simonl9l]

@norm any next steps here this is really killing us.

We have developers on M1 Apple Silicon, that are unable to cross compile (via Docker) to linux/x86_64 and the docker/qemu environment does not currently support it at all due to multiple layers of emulation.

[normj]

@Simonl9l sorry no progress on this issue. We have been in conversation with Microsoft on the issue discussing possible solutions but as they are heads down getting .NET 7 out the door we don't expect much traction on this issue till post .NET 7 release.

[tmitchel2]

Would really like to second this. Getting AL2022 on lambda ASAP seems like the right way to go but if any workaround is possible I would love to hear it.

[SamuelCox]

Any update on this post dotnet7 release?

[tyler-shuhnicki]

I'm also wondering - is there any update on this issue? I understand this is a lambda specific forum, but I'm having the same issue deploying dotnet 7 to an Elastic Beanstalk / EC2 instance on ARM64.

[erwan-joly]

I'm also wondering - is there any update on this issue? I understand this is a lambda specific forum, but I'm having the same issue deploying dotnet 7 to an Elastic Beanstalk / EC2 instance on ARM64.

From @normj https://github.com/aws/aws-lambda-dotnet/issues/1416#issuecomment-1405348027 It seems like there are good progress made on this 🎉

[purkhusid]

Will we need to wait for .Net 7.0.4?

[martincostello]

I've just updated the AWS Lambda function in the original issue description to .NET 7.0.4 (https://github.com/martincostello/alexa-london-travel/commit/fba0da0683883f31b01c3daa2249dcc745ec37df) and changed it back to arm64 from x64 (https://github.com/martincostello/alexa-london-travel/commit/8f689eb72a796c1b79ee2f052b5cf72ec5299da5) and it's now working.

Thanks to the AWS and Microsoft teams for their efforts on getting things working. 🙇

[normj]

Lots of thanks and appreciation to the .NET team for making the changes needed to run .NET 7 ARM on Amazon Linux 2.

[AbbTek]

Cool... we also updated back to arm64... working perfectly... thanksssss

[ashishdhingra]

@martincostello Please confirm if we could close the issue given that .NET 7 ARM on AL2 is working for you.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.