aws
Bug: docker with aws' `sam build` on M1 for amd64 cannot build
Disclaimer
This issue is merely for reference purposes. It's not necessarily intended to be used as triaging for fixing.
Reason: This might be a difficult edge case and I don't have a from-scratch-like clean environment to make sure the steps will lead to a failure.
Description:
Docker build fails when triggered by AWS' sam build on Mac M1 with target amd64 (cross-plattform).
Error: Build Failed Error: NotFound: content digest sha256:[...]: not found
Notes:
- The build succeeds if it's directly invoked by
docker build. - The build succeeds if the target is arm64
- It's not clear whether this is a sam problem or a docker problem.
Workaround stated at the bottom.
Steps to reproduce:
- Configuration
- AWS SAM version: version 1.125.0
- Host: Mac M1
- Docker 4.34.3 or any other version until 4.29
- Target platform: amd64
- buildkit: activated
Files
*template.yaml`
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: asdf
Resources:
HjksDataCrawler:
Type: AWS::Serverless::Function
Properties:
Description: function
FunctionName: AFunction
PackageType: Image
Architectures:
- x86_64
Metadata:
Dockerfile: Dockerfile
DockerContext: ./
Dockerfile
FROM --platform=linux/amd64 public.ecr.aws/lambda/python:3.11
COPY app.py /var/task
CMD [ "app.lambda_handler" ]
(↑ I hope I didn't strip too much)
app.py
def lambda_handler():
print("hello world")
samconfig.toml
version = 0.1
[dev]
[dev.build]
[dev.build.parameters]
region="ap-east-1"
skip_pull_image=true
use_container=true
template = "template.yml"
base_dir = "."
# s3_bucketやimage_repositoryは仮置き
[dev.deploy]
[dev.deploy.parameters]
stack_name = "teststack"
image_repository = # TODO: NEEDS TO BE FILLED IN
region = "ap-east-1"
confirm_changeset = true
capabilities = "CAPABILITY_IAM"
s3_bucket = # TODO: NEEDS TO BE FILLED IN
s3_prefix = "docker_bug_test_m1_x86"
Command to run
export DOCKER_HOST=unix://"$HOME/.docker/run/docker.sock"
sam build --config-env dev
Note:
- For the DOCKER_HOST setting you can also use this more cryptic command, which leads to the same result
Observed result:
Error output: Build Failed Error: NotFound: content digest sha256:[...]: not found
I can't reproduce the debug output, because I already workaround-ed the problem. 🙏
Expected result:
Build should succeed.
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
- OS: macOS Sonoma 14.5 (14.5 (23F79))
-
sam --version: version 1.125.0 - AWS region: ap-east-1
- Docker 4.34.3 or any other version until 4.29
sam --info
{
"version": "1.125.0",
"system": {
"python": "3.12.6",
"os": "macOS-14.5-arm64-arm-64bit"
},
"additional_dependencies": {
"docker_engine": "20.10.14",
"aws_cdk": "2.61.1 (build d319d9c)",
"terraform": "Not available"
},
"available_beta_feature_env_vars": [
"SAM_CLI_BETA_FEATURES",
"SAM_CLI_BETA_BUILD_PERFORMANCE",
"SAM_CLI_BETA_TERRAFORM_SUPPORT",
"SAM_CLI_BETA_RUST_CARGO_LAMBDA"
]
}
Workaround
Downgrade to Docker 4.7, downloadable here: https://desktop.docker.com/mac/main/arm64/77141/Docker.dmg ( @Docker maintainers: Please don't remove the download link in order to keep the workaround working 🙏 )
Hi @stationname, thanks for reporting the issue and providing how you managed to get it to work. Marking it as needs more investigation to see if this is a sam or docker issue.
Example debug output
2024-12-24 01:22:38,708 | Config file location: /Users/arvindh/cs/projects/kp-restructure/lmsys/samconfig.toml
2024-12-24 01:22:38,709 | Loading configuration values from [default.['build'].parameters] (env.command_name.section) in config file at
'/Users/arvindh/cs/projects/kp-restructure/lmsys/samconfig.toml'...
2024-12-24 01:22:38,710 | Configuration values successfully loaded.
2024-12-24 01:22:38,710 | Configuration values are: {'stack_name': 'lmsys', 'parallel': True}
2024-12-24 01:22:38,713 | Using SAM Template at /Users/arvindh/cs/projects/kp-restructure/lmsys/template.yaml
2024-12-24 01:22:38,732 | Using config file: samconfig.toml, config environment: default
2024-12-24 01:22:38,732 | Expand command line arguments to:
2024-12-24 01:22:38,732 | --template_file=/Users/arvindh/cs/projects/kp-restructure/lmsys/template.yaml --parallel --mount_with=READ
--build_dir=.aws-sam/build --cache_dir=.aws-sam/cache
2024-12-24 01:22:38,757 | 'build' command is called
2024-12-24 01:22:38,759 | No Parameters detected in the template
2024-12-24 01:22:38,771 | There is no customer defined id or cdk path defined for resource ${Redacted}, so we will use the resource logical id
as the resource id
2024-12-24 01:22:38,771 | There is no customer defined id or cdk path defined for resource ${Redacted}, so we will use the resource logical
id as the resource id
2024-12-24 01:22:38,772 | 0 stacks found in the template
2024-12-24 01:22:38,772 | No Parameters detected in the template
2024-12-24 01:22:38,781 | There is no customer defined id or cdk path defined for resource ${Redacted}, so we will use the resource logical id
as the resource id
2024-12-24 01:22:38,781 | There is no customer defined id or cdk path defined for resource ${Redacted}, so we will use the resource logical
id as the resource id
2024-12-24 01:22:38,782 | 2 resources found in the stack
2024-12-24 01:22:38,783 | Found Serverless function with name='${Redacted}' and ImageUri='None'
2024-12-24 01:22:38,783 | --base-dir is not presented, adjusting uri ./${Redacted}-lambda relative to
/Users/arvindh/cs/projects/kp-restructure/lmsys/template.yaml
2024-12-24 01:22:38,784 | --base-dir is not presented, adjusting uri . relative to /Users/arvindh/cs/projects/kp-restructure/lmsys/template.yaml
2024-12-24 01:22:38,787 | 2 resources found in the stack
2024-12-24 01:22:38,788 | Found Serverless function with name='${Redacted}' and ImageUri='None'
2024-12-24 01:22:38,788 | Error occurred while trying to track an event: Event 'BuildFunctionRuntime' does not accept value 'None'.
2024-12-24 01:22:38,789 | Instantiating build definitions
2024-12-24 01:22:38,791 | Unique function build definition found, adding as new (Function Build Definition: BuildDefinition(None,
/Users/arvindh/cs/projects/kp-restructure/lmsys, Image, , b8e87180-e974-4d8d-b2e9-5d09841c1aa4, {'DockerTag': 'provided.al2023-v1', 'DockerContext':
'/Users/arvindh/cs/projects/kp-restructure/lmsys/lmsys-${Redacted}-lambda', 'Dockerfile': 'Dockerfile'}, {}, x86_64, []), Function: Function({'function_id':
'${Redacted}', 'name': '${Redacted}', 'functionname': '${Redacted}', 'runtime': None, 'memory': 128, 'timeout': 5, 'handler':
None, 'imageuri': None, 'packagetype': 'Image', 'imageconfig': None, 'codeuri': '/Users/arvindh/cs/projects/kp-restructure/lmsys', 'environment': None,
'rolearn': 'arn:aws:lambda:us-east-1:123456789012:function:${Redacted}', 'layers': [], 'events': {'QueueScheduler': {'Type': 'Schedule',
'Properties': {'Schedule': 'rate(5 minutes)', 'Enabled': True, 'Name': '${Redacted}', 'Description': 'Schedule to trigger
${Redacted} every 5 minutes'}}}, 'metadata': {'DockerTag': 'provided.al2023-v1', 'DockerContext':
'/Users/arvindh/cs/projects/kp-restructure/lmsys/${Redacted}-lambda', 'Dockerfile': 'Dockerfile', 'SamResourceId': '${Redacted}'},
'inlinecode': None, 'codesign_config_arn': None, 'architectures': ['x86_64'], 'function_url_config': None, 'function_build_info':
<FunctionBuildInfo.BuildableImage: ('BuildableImage', 'Regular IMAGE function which can be build with SAM CLI')>, 'stack_path': '',
'runtime_management_config': None, 'logging_config': None}))
2024-12-24 01:22:38,810 | Async execution started
2024-12-24 01:22:38,810 | Invoking function functools.partial(<bound method ParallelBuildStrategy.build_single_function_definition of
<samcli.lib.build.build_strategy.ParallelBuildStrategy object at 0x103c22090>>, <samcli.lib.build.build_graph.FunctionBuildDefinition object at
0x10684b410>)
2024-12-24 01:22:38,811 | Building codeuri: /Users/arvindh/cs/projects/kp-restructure/lmsys runtime: None architecture: x86_64 functions:
${Redacted}
2024-12-24 01:22:38,811 | Building to following folder /Users/arvindh/cs/projects/kp-restructure/lmsys/.aws-sam/build/${Redacted}
2024-12-24 01:22:38,811 | Building image for ${Redacted} function
2024-12-24 01:22:38,812 | Waiting for async results
2024-12-24 01:22:38,816 | Setting DockerBuildArgs for ${Redacted} function
2024-12-24 01:22:40,307 | Failed building function ${Redacted}
2024-12-24 01:22:40,310 | Exception raised during the execution
Build Failed
2024-12-24 01:22:40,311 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-12-24 01:22:40,371 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-12-24 01:22:40,372 | Unable to find Click Context for getting session_id.
2024-12-24 01:22:40,372 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '7b1de4c5-37ea-4c50-a6f4-c97797106dfc', 'installationId':
'55c6fc5a-513a-4288-a9a2-475796b2a85c', 'sessionId': 'f87faa4b-c244-4fef-a67f-f454731481d4', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion':
'3.11.10', 'samcliVersion': '1.132.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build',
'metricSpecificAttributes': {'projectType': 'CFN', 'gitOrigin': None, 'projectName': '636fa9664f878a4de79227421c3c5199b1d7ced700dc2171f7e22d4253accd25',
'initialCommit': None}, 'duration': 1579, 'exitReason': 'DockerBuildFailed', 'exitCode': 1}}]}
2024-12-24 01:22:40,375 | Sending Telemetry: {'metrics': [{'events': {'requestId': '64d09680-a58a-49e4-a99a-13f112d2ffa4', 'installationId':
'55c6fc5a-513a-4288-a9a2-475796b2a85c', 'sessionId': 'f87faa4b-c244-4fef-a67f-f454731481d4', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion':
'3.11.10', 'samcliVersion': '1.132.0', 'commandName': 'sam build', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension',
'event_value': '.toml', 'thread_id': 'a8fa6c4f6c584fe084a942f01640b64f', 'time_stamp': '2024-12-24 06:22:38.708', 'exception_name': None},
{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '27b2a0b86cae42e6b020ea0980ac17ac', 'time_stamp': '2024-12-24
06:22:38.731', 'exception_name': None}]}}}]}
2024-12-24 01:22:40,690 | Telemetry response: 200
2024-12-24 01:22:40,693 | Telemetry response: 200
Error: failed to get destination image "sha256:8de8595beec4ae7521d32e3527054f4336b10f0e4246ec7b0cf6584702b7ae7b": No such image: sha256:8de8595beec4ae7521d32e3527054f4336b10f0e4246ec7b0cf6584702b7ae7b
similar issue on M4 mac. company provided me a new laptop and ive been racking my brain for a few hours trying to figure out why i cant build & deploy this SAM project.
thank you for the workaround
Thanks for the work around, it really worked for me. Please take a look at this issue, within Mac mini M4 happened exactly with the same debug output as above. Sequoia 15.2.
+1 MB air M2
Docker 4.19.0
Build Failed
2025-02-23 11:30:16,826 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2025-02-23 11:30:16,865 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2025-02-23 11:30:16,866 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'b1485a20-cb72-465f-937b-cdc9cb45c67d', 'installationId': '3c9de2d4-85e2-4502-ac86-b5ff47913959', 'sessionId': '2538239a-f0a9-416c-adf4-3c236fffc2d7',
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.11.10', 'samcliVersion': '1.134.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'metricSpecificAttributes': {'projectType':
'CFN', 'gitOrigin': None, 'projectName': '593ab2ca51e925b9f6c2f258bc55ed5926cf6d2c78239a685d65907e4ec7edd3', 'initialCommit': None}, 'duration': 71, 'exitReason': 'DockerConnectionError', 'exitCode': 1}}]}
2025-02-23 11:30:16,866 | Unable to find Click Context for getting session_id.
2025-02-23 11:30:16,868 | Sending Telemetry: {'metrics': [{'events': {'requestId': '58413bb5-102b-475a-8688-20f4d444e772', 'installationId': '3c9de2d4-85e2-4502-ac86-b5ff47913959', 'sessionId': '2538239a-f0a9-416c-adf4-3c236fffc2d7',
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.11.10', 'samcliVersion': '1.134.0', 'commandName': 'sam build', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml',
'thread_id': 'ed4f6d3c543840269c38c0fe0109b4de', 'time_stamp': '2025-02-23 04:30:16.735', 'exception_name': None}, {'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': 'af0eb6e845e94af4863455b0966f5e41',
'time_stamp': '2025-02-23 04:30:16.755', 'exception_name': None}]}}}]}
2025-02-23 11:30:17,400 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
2025-02-23 11:30:17,403 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Solved with Docker 4.37.2 and AWS SAM 1.124
To install an older version of AWS SAM CLI via the command line, I used pipx as follows:
brew install pipx
pipx ensurepath
pipx install aws-sam-cli==1.124.0
pipx ensurepath
Additionally, I downgraded to Docker 4.37.2 and, most importantly, performed a “Reset to factory defaults” after installation.
This setup finally got everything working for me.
In my case, the Lambda base image public.ecr.aws/lambda/nodejs:22 wasn't properly available locally (I sam initwith nodejs 22).
failed to export image: NotFound: content digest sha256:a65409354cd6218bcff890b9efd4a6e3ea02aa62d847bb5a22a65de00546dd82: not found
I explicitly pulled the Base Image and rebuilt. Works for me:
docker pull public.ecr.aws/lambda/nodejs:22
