sam build: Optional flag to preserve CodeUri parent directory

[heitorlessa]

Describe your idea/feature/enhancement

For Python applications, it'd be ideal to have a flag in sam build to preserve the parent folder in CodeUri when building and copying artifacts to $ARTIFACTS_DIR.

For example, I wish SAM CLI could allow Python developers to keep their layout for modules like we do elsewhere:

Ideal

.
├── loyalty
│   └── __init__.py
│   └── aggregate
│       ├── __init__.py
│       └── app.py
│   └── get
│       ├── __init__.py
│       └── app.py
│   └── ingest
│       ├── __init__.py
│       └── app.py
│   └── shared
│       ├── __init__.py
│       ├── constants.py
│       ├── functions.py
│       ├── models.py
│       └── storage.py
├── template.yaml
└── tests
    ├── conftest.py
    ├── events
    │   ├── aggregate_insert_event.json
    │   ├── get_event.json
    │   └── ingest_event.json
    ├── storage
    │   ├── __init__.py
    │   └── test_dynamodb.py
    └── unit
        ├── __init__.py
        ├── test_aggregate.py
        ├── test_get_loyalty_points.py
        └── test_ingest.py

Template

...
  IngestFunc:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: !Sub ServerlessAirline-IngestLoyalty-${Stage}
      CodeUri: loyalty
      Handler: loyalty.ingest.app.lambda_handler

Import line as well as tests

from loyalty.shared.models import LoyaltyPoint, LoyaltyPointAggregate
from loyalty.shared.storage import FakeStorage, DynamoDBStorage
...

To workaround this issue, I have to create another folder src to get the intended result[1], but this creates other problems in PYTHONPATH which I need to adjust in my IDE and tools later. Alternatively, I could use BuildMethod: makefile in SAM, but then I to lose dedup functionality.

Steps to reproduce the issue:

1. Use loyalty as CodeUri
2. Run sam build

Observed result: loyalty parent folder will not be copied over but its contents

Expected result: A flag to ensure folder in CodeUri is preserved when building artifacts, e.g. --preserve-, or perhaps respect foldervsfolder/`.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc) Mac

Implementation in question that deliberately removes the parent folder: https://github.com/aws/aws-lambda-builders/blob/41804217a18ff47cce5b62eff5cf884a7f81a0c5/aws_lambda_builders/utils.py#L16

Lambda Builders related issue: https://github.com/aws/aws-lambda-builders/issues/283

---

[1] Current layout to ensure loyalty folder is copied over (CodeUri: src):

.
├── src
│   └── loyalty
│       ├── __init__.py
│       ├── aggregate
│       │   ├── __init__.py
│       │   └── app.py
│       ├── get
│       │   ├── __init__.py
│       │   └── app.py
│       ├── ingest
│       │   ├── __init__.py
│       │   └── app.py
│       └── shared
│           ├── __init__.py
│           ├── constants.py
│           ├── functions.py
│           ├── models.py
│           └── storage.py
├── template.yaml
└── tests
    ├── conftest.py
    ├── events
    │   ├── aggregate_insert_event.json
    │   ├── get_event.json
    │   └── ingest_event.json
    ├── storage
    │   ├── __init__.py
    │   └── test_dynamodb.py
    └── unit
        ├── __init__.py
        ├── test_aggregate.py
        ├── test_get_loyalty_points.py
        └── test_ingest.py

Proposal

A new CLI flag --preserve-code-uri, --preserve, etc.

Things to consider:

1. Will this require any updates to the SAM Spec

Additional Details

[CoshUS]

Thanks for the feedback. We will look into this proposal.

[JJSphar]

The behavior of the codeUri property makes it challenging to use SAM with multiple Python Lambda functions/layers in one repo. Here's an abstract example:

I have a repository project that should deploy a Lambda Function and a Lambda layer. My python code is broken into packages function and layer.

This is how I want to structure my project:

project/
├─ src/
│  ├─ function/
│  │  ├─ __init__.py
│  │  ├─ module_a.py
│  │  ├─ module_b.py
│  │  ├─ requirements.txt
│  ├─ layer/
│  │  ├─ __init__.py
│  │  ├─ ...
│  │  ├─ requirements.txt
├─ setup.py  # allow the `layer` package to be pip installed
├─ template.yaml
├─ samconfig.toml

The benefits I get with this layout:

• All source code is centralized under src/. This keeps the packages off the PATH by default, and forces me to test the installed code.
• I can use layer in a future project: Add the layer version to the new lambda, and install the package for development using pip install -e file://path/to/project
• A common/popular layout that's familiar to other Python developers/tools/libraries.

However, this layout cannot be used if you want SAM to package and deploy your resources. Three examples:

Many people's first instinct is to set the CodeUri and ContentUri properties to be function and layer, which causes deployed code to fail:

• Since only the contents of the CodeUri folder are copied over, imports are broken
• Writing from function.module_b import method in module_a no longer works - because the two modules are no longer inside a folder named function

Another possibility would be setting both CodeUris to the src/ folder. I don't like this, because all the code gets packaged into both resources, which defeats the point of using layers.

The solution mentioned by the author is that each package needs to go in its own folder with nothing else: project/ ├─ FunctionSrc/ │ ├─ function/ │ │ ├─ ... ├─ LayerSrc/ │ ├─ layer/ │ │ ├─ ... ├─ ...

Where you then set the CodeUri to FunctionSrc/LayerSrc. This is how I've been structuring my projects for the last 6 months or so, and it's been workable at best.

This 'separated' layout causes extra headaches when working with standard python tools. Since the source is no longer contained in one place, every new function/layer you add means changing your PYTHONPATH, or adding a new entry to the package_dir field in setup.py. My current issue is that this layout is incompatible with a pip install -e command:

I am trying to write a "HelperLayers" repository that publishes multiple lambda layers. Each layer's package is in it's own "LayerSrc" folder. I want to make my IDE aware of the helper code by pip install -e, but multiple directory:package mappings in a setup.py are not supported for editable installs.

It seems like the --preserve-code-uri option would let me use the "first instinct" approach (each codeUri can be set to the package) without breaking any imports when deployed. And, since all the packages are under a single folder, I can write package_dir={"": "src"} in my setup.py to make the whole thing work with pip for local development. Perfect!

What are other people doing for structuring more complicated SAM projects? I'm open to suggestions on how to make both SAM and pip/pycharm/pytest happy.

Thanks for reading - and thanks for your writeup, @heitorlessa. By the way, I'm a big fan of the aws-lambda-powertools package :)

[heitorlessa]

Cc @jfuss @praneetap @awood45 - I know the team is crazy busy but if you're happy with the idea, I'm happy to create a PR for you to review and suggest a better UX.

This will make Python experience for newcomers and micro functions a problem solved.

[heitorlessa]

Checking in as I've had to help a customer create multiple workarounds in VSCode for this to work (eg., tests failing due to module not found, FQDN imports failing in Lambda because parent folder is deleted when packaging)

[hampsterx]

sorry to +1 but this is really annoying for python developers coming into AWS SAM.

• my_module
• tests
• template.yaml

This structure is what you would expect, period. But no, we are forced to find some kind of PITA workaround :(

[reedham-aws]

I believe this issue has been fixed by #8248, closing as complete.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.