feat: Support installing poetry dependencies with pip

[pdecat]

Description

This PR adds support for installing poetry dependencies with pip --no-deps for projects with pyproject.toml and optionally poetry.lock files.

We use pip as as poetry does not currently provide a proper way to isolate installed dependencies from virtualenv (pip's --target option). --no-deps disables pip's dependency resolver as poetry already pinned all dependencies.

This is a continuation of https://github.com/terraform-aws-modules/terraform-aws-lambda/pull/186

Motivation and Context

poetry's is more and more used and has a robust dependency resolver.

Breaking Changes

New poetry install step is not enabled by default and requires explicitly setting poetry_install = True in the source block.

How Has This Been Tested?

• [x] I have updated at least one of the examples/* to demonstrate and validate my change(s)
• [x] I have tested and validated these changes using one or more of the provided examples/* projects

• [x] I have executed pre-commit run -a on my pull request

cd examples/build-package
terraform init
rm -rf builds/lambda_layer/ && terraform apply -target module.lambda_layer -auto-approve
rm -rf builds/lambda_layer_poetry/ && terraform apply -target module.lambda_layer_poetry -auto-approve
rm -rf builds/package_dir/ && terraform apply -target module.package_dir -auto-approve
rm -rf builds/package_dir_poetry/ && terraform apply -target module.package_dir_poetry -auto-approve
terraform destroy

[github-actions[bot]]

This PR has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this PR will be closed in 10 days

[pdecat]

Still current

[github-actions[bot]]

This PR has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this PR will be closed in 10 days

[pdecat]

Still current

[1davidmichael]

I primarily use poetry with Python and appreciate this PR. I hope it is merged at some point in the future. Thanks for this work!

[antonbabenko]

@pdecat Please let us know if/when this PR is ready for review and merge?

I expect that other users (including @1davidmichael @panessa @Tonkonozhenko) will be able to help with the review and testing.

[pdecat]

Hi, I believe it is ready for review and testing by others.

[pdecat]

Hi @antonbabenko, some Github Actions jobs failed because of network errors. Mind to restart them? :pray:

[antonbabenko]

@pdecat Restarted GitHub Actions, and all checks are green. I have scheduled to revive this PR on the 13th of October (after HashiConf). Hopefully, everyone else will be addressing all the comments and do a review, too.

[pdecat]

I just tried to test it locally after terraform-linters/tflint#362 was merged, and apparently, it broke package.py you've created.

The error message I got during terraform destroy after the successful run of rm -rf builds/lambda_layer/ && terraform apply -target module.lambda_layer:

│     npm_requirements_step(
│   File ".../terraform-aws-modules/terraform-aws-lambda/examples/build-package/../../package.py", line 709, in npm_requirements_step
│     raise RuntimeError(
│ RuntimeError: Nodejs interpreter version equal to
│ defined lambda runtime (nodejs14.x) should be available
│ in system PATH
│ 
│ State: exit status 1

The same errors happen using current master branch for all examples that do not have build_in_docker = true for which the configured runtime is not available locally.

I see at least 3 solutions:

1. install all runtimes used in the examples
2. update all examples to only use locally available runtimes
3. remove the runtime checks for the examples

As a work-around for now, only destroy the modules that were applied with resource targeting:

terraform destroy -target module.lambda_layer
terraform destroy -target module.lambda_layer_poetry
terraform destroy -target module.package_dir
terraform destroy -target module.package_dir_poetry

[pdecat]

The Pre-Commit / Max TF pre-commit check is failing with a weird error:

Terraform validate with tflint...........................................Failed
- hook id: terraform_tflint
- exit code: 1

Command 'tflint --init' successfully done:




TFLint in modules/deploy/:
Failed to check ruleset; Failed to check `terraform_deprecated_index` rule: main.tf:52,8-9: Invalid character; This character is not used within the language., and 57 other diagnostic(s)

See https://github.com/terraform-aws-modules/terraform-aws-lambda/actions/runs/3312065827/jobs/5468288824

And does not only affect jobs from this PR, other PRs are also affected, see https://github.com/terraform-aws-modules/terraform-aws-lambda/actions/runs/3313158688/jobs/5470763712

Edit: this seems to be caused by https://github.com/terraform-linters/tflint-ruleset-terraform/issues/42

[pdecat]

Pre-commit hooks should be fixed now: https://github.com/terraform-linters/tflint/releases/tag/v0.42.2

[antonbabenko]

This PR is included in version 4.3.0 :tada:

[Tonkonozhenko]

@pdecat @antonbabenko thank you for the work! we'll use it asap

[github-actions[bot]]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

[antonbabenko]

This issue has been resolved in version 4.16.0 :tada: