aws
Bug: sam local start-api fails to initialise lambda function containers within podman/toolbox container
Description:
When attempting to run sam local start-api from within podman/toolbox on Fedora 40 Kinoite, the lambda function containers fail to initialise. Command hangs for over an hour (at least) without initialising containers. On one occassion, when left overnight it initialised the container and was ready to be called.
To test, utilised AWS Lambda powertools tutorial template found here: https://docs.powertools.aws.dev/lambda/python/latest/tutorial/#code-example
Output Initializing the lambda functions containers.
Steps to reproduce:
Fedora 40 Kinoite -> Toolbox
toolbox create sam-test
toolbox enter sam-test
cd temp
sudo ./sam-installation/install
export DOCKER_HOST=unix:///run/user/1000/podman/podman.sock
sam build && sam local start-api --debug
Observed result:
[user@toolbox hello-world-tutorial]$ sam build && sam local start-api --debug
Starting Build use cache
Manifest is not changed for (HelloWorldFunction), running incremental build
Building codeuri: /var/home/user/hello-world-tutorial/hello_world runtime: python3.12 metadata: {} architecture: x86_64
functions: HelloWorldFunction
Running PythonPipBuilder:CopySource
Running PythonPipBuilder:CopySource
Build Succeeded
Built Artifacts : .aws-sam/build
Built Template : .aws-sam/build/template.yaml
Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke
[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch
[*] Deploy: sam deploy --guided
2024-08-01 13:57:06,971 | Config file location: /var/home/user/hello-world-tutorial/samconfig.toml
2024-08-01 13:57:06,973 | Loading configuration values from [default.['local', 'start-api'].parameters] (env.command_name.section) in
config file at '/var/home/user/hello-world-tutorial/samconfig.toml'...
2024-08-01 13:57:06,974 | Configuration values successfully loaded.
2024-08-01 13:57:06,975 | Configuration values are: {'stack_name': 'hello-world-tutorial', 'warm_containers': 'EAGER'}
2024-08-01 13:57:06,978 | Using SAM Template at /var/home/user/hello-world-tutorial/.aws-sam/build/template.yaml
2024-08-01 13:57:07,041 | Using config file: samconfig.toml, config environment: default
2024-08-01 13:57:07,042 | Expand command line arguments to:
2024-08-01 13:57:07,042 | --template_file=/var/home/user/hello-world-tutorial/.aws-sam/build/template.yaml
--host=127.0.0.1 --port=3000 --static_dir=public --layer_cache_basedir=/home/user/.aws-sam/layers-pkg --warm_containers=EAGER
--container_host=localhost --container_host_interface=127.0.0.1
2024-08-01 13:57:07,097 | local start-api command is called
2024-08-01 13:57:08,114 | No Parameters detected in the template
2024-08-01 13:57:08,159 | Sam customer defined id is more priority than other IDs. Customer defined id for resource HelloWorldFunction
is HelloWorldFunction
2024-08-01 13:57:08,160 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the
resource logical id as the resource id
2024-08-01 13:57:08,162 | 0 stacks found in the template
2024-08-01 13:57:08,163 | No Parameters detected in the template
2024-08-01 13:57:08,185 | Sam customer defined id is more priority than other IDs. Customer defined id for resource HelloWorldFunction
is HelloWorldFunction
2024-08-01 13:57:08,186 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the
resource logical id as the resource id
2024-08-01 13:57:08,187 | 2 resources found in the stack
2024-08-01 13:57:08,188 | Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'
2024-08-01 13:57:08,189 | --base-dir is not presented, adjusting uri HelloWorldFunction relative to
/var/home/user/hello-world-tutorial/.aws-sam/build/template.yaml
2024-08-01 13:57:08,191 | watch resource /var/home/user/hello-world-tutorial/.aws-sam/build/template.yaml
2024-08-01 13:57:08,192 | Create Observer for resource /var/home/user/hello-world-tutorial/.aws-sam/build/template.yaml
with recursive True
2024-08-01 13:57:08,193 | watch resource /var/home/user/hello-world-tutorial/.aws-sam/build/template.yaml's parent
/var/home/user/hello-world-tutorial/.aws-sam/build
2024-08-01 13:57:08,194 | Create Observer for resource /var/home/user/hello-world-tutorial/.aws-sam/build with recursive
False
2024-08-01 13:57:08,200 | Initializing the lambda functions containers.
2024-08-01 13:57:08,202 | Async execution started
2024-08-01 13:57:08,203 | Invoking function functools.partial(<function
InvokeContext._initialize_all_functions_containers.<locals>.initialize_function_container at 0x7f23bff07920>,
Function(function_id='HelloWorldFunction', name='HelloWorldFunction', functionname='HelloWorldFunction', runtime='python3.12',
memory=None, timeout=3, handler='app.lambda_handler', imageuri=None, packagetype='Zip', imageconfig=None,
codeuri='/var/home/user/hello-world-tutorial/.aws-sam/build/HelloWorldFunction', environment=None, rolearn=None,
layers=[], events={'HelloWorld': {'Type': 'Api', 'Properties': {'Path': '/hello', 'Method': 'get', 'RestApiId':
'ServerlessRestApi'}}}, metadata={'SamResourceId': 'HelloWorldFunction'}, inlinecode=None, codesign_config_arn=None,
architectures=['x86_64'], function_url_config=None, function_build_info=<FunctionBuildInfo.BuildableZip: ('BuildableZip', 'Regular ZIP
function which can be build with SAM CLI')>, stack_path='', runtime_management_config=None, logging_config=None))
2024-08-01 13:57:08,208 | Waiting for async results
Expected result:
Building image..........................
2024-08-01 13:32:44 WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
* Running on http://127.0.0.1:3000
* 2024-08-01 13:32:44 Press CTRL+C to quit
Expected Lambda to initialise and be available.
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
OS: Fedora 40 Kinoite { "version": "1.120.0", "system": { "python": "3.11.8", "os": "Linux-6.9.10-200.fc40.x86_64-x86_64-with-glibc2.39" }, "additional_dependencies": { "docker_engine": "5.1.2", "aws_cdk": "Not available", "terraform": "Not available" }, "available_beta_feature_env_vars": [ "SAM_CLI_BETA_FEATURES", "SAM_CLI_BETA_BUILD_PERFORMANCE", "SAM_CLI_BETA_TERRAFORM_SUPPORT", "SAM_CLI_BETA_RUST_CARGO_LAMBDA" ] }
Hi @AidanOB, thanks for reporting the issue and providing details. I will try to reproduce the issue.
Can you check if sam local invoke is able to initialize the containers for you and invoke the lambdas?
Unfortunately the containers don't initialise either. Regular output is just no response on the terminal.
With the -debug flag set:
2024-08-06 10:49:37,750 | Config file location: /var/home/user/hello-world-tutorial/samconfig.toml
2024-08-06 10:49:37,753 | Loading configuration values from [default.['local', 'invoke'].parameters] (env.command_name.section) in
config file at '/var/home/user/hello-world-tutorial/samconfig.toml'...
2024-08-06 10:49:37,754 | Configuration values successfully loaded.
2024-08-06 10:49:37,754 | Configuration values are: {'stack_name': 'hello-world-tutorial'}
2024-08-06 10:49:37,757 | Using SAM Template at /var/home/user/hello-world-tutorial/.aws-sam/build/template.yaml
2024-08-06 10:49:37,834 | Using config file: samconfig.toml, config environment: default
2024-08-06 10:49:37,835 | Expand command line arguments to:
2024-08-06 10:49:37,835 | --template_file=/var/home/user/hello-world-tutorial/.aws-sam/build/template.yaml --no_event
--layer_cache_basedir=/home/user/.aws-sam/layers-pkg --container_host=localhost --container_host_interface=127.0.0.1
2024-08-06 10:49:37,836 | local invoke command is called
2024-08-06 10:49:39,126 | No Parameters detected in the template
2024-08-06 10:49:39,168 | Sam customer defined id is more priority than other IDs. Customer defined id for resource
HelloWorldFunction is HelloWorldFunction
2024-08-06 10:49:39,170 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the
resource logical id as the resource id
2024-08-06 10:49:39,173 | 0 stacks found in the template
2024-08-06 10:49:39,174 | No Parameters detected in the template
2024-08-06 10:49:39,204 | Sam customer defined id is more priority than other IDs. Customer defined id for resource
HelloWorldFunction is HelloWorldFunction
2024-08-06 10:49:39,206 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the
resource logical id as the resource id
2024-08-06 10:49:39,208 | 2 resources found in the stack
2024-08-06 10:49:39,209 | Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'
2024-08-06 10:49:39,211 | --base-dir is not presented, adjusting uri HelloWorldFunction relative to
/var/home/user/hello-world-tutorial/.aws-sam/build/template.yaml
Further behaviour I have discovered, but intermittently (so unable to reliably reproduce) is that the lambdas will initialise if I do other things with toolbox/podman. Sometimes making or entering a new toolbox will trigger the initialisation of the lambdas. Or making another container. But this has only happened once or twice and not exactly sure what I did at the time that caused the lambdas in the other container to suddenly initialise.
I'm experiencing something similar.
The sam commands (either sam local start-api or sam local invoke) simply hang indefinitely, until I would run something like podman run hello-world in a separate shell.
Somehow this allows the sam process to continue and everything works as expected.
Environment details: podman version 5.2.0 macOS Sonoma Version 14.5 SAM CLI, version 1.121.0
Thanks @gergo-dryrun - running podman run hello-world in another shell worked!
Attempt to fix this in #7760
Thank you! The changes there fixed the issue for me. Looking forward to seeing that released!
@gergo-dryrun You sir deserve more likes! Thank you!!! Now only if we know why your solution works would be even better 😄 🚀
@gminova It's because the event SAM cli is reading from blocks until there's an event of type image triggered, which happens when you run a pod or pull an image.
@chiting Thank you. Yes, I saw the reason described in your PR, hopefully you can get your fix merged, however I wonder why sam is working seamlessly with docker and does not have the same problem whereas with podman this issue happens. Perhaps there could be some fix podman could do instead of sam?
@gminova I agree with you, there seem to be some differences around events and it would be good for podman to close the gap with how docker behaves.
Regardless of the above, I still believe SAM cli shouldn't use blocking calls in the main thread (unless it's by design, which I could be missing). The fact it works with docker seems to be server side implementation specific and I think the tool shouldn't rely on it to work.
@chiting Hey, I was able to fix my issue locally by removing my machine and init it again and ensuring the podman info is using version 5.3 where they have a fix, please see issue and discussion here: https://github.com/containers/podman/issues/24839#issuecomment-2544290005
I think this issue can be closed @AidanOB
Hi @gminova, upgrading my podman to 5.3 works for me! you're just saving my day, thank you.
