aws
sam build does not recognise local nodejs projects outside CodeUri
Description
I have 2 Node.js projects: CommonT and Profile. Profile has a local dependency on CommonT.
When running the sam build -t template.yaml --debug command, it fails with the following error:
npm ERR! code ENOLOCAL
npm ERR! Could not install from "../CommonT" as it does not contain a package.json file.
Here is the project structure:
├── Backend
│ ├── CommonT
│ │ ├── package.json
│ │ └── src
│ └── Profile
│ ├── package.json
│ └── src
└── template.yaml
Here is the Profile/package.json:
{
"main": "dist/src/index.js",
...
"scripts": {
"test": "ts-node ./node_modules/jasmine/bin/jasmine",
"build": "tsc",
"deploy": "npm run build && sls deploy"
},
"dependencies": {
"@myteam/common": "file:../CommonT",
...
},
"devDependencies": {
"@types/aws-lambda": "^8.10.46",
"@types/node": "^12.12.32",
"aws-sdk": "^2.508.0",
"ts-node": "^8.10.2",
"typescript": "^3.8.3",
...
}
}
Here is the CommonT/package.json:
{
"name": "@myteam/common",
...
"main": "dist/src/index",
"typings": "dist/src/index",
"private": true,
"scripts": {
"test": "ts-node ./node_modules/jasmine/bin/jasmine",
"build": "tsc",
"deploy": "npm run build && sls deploy"
},
"dependencies": {
...
},
"devDependencies": {
"@types/aws-lambda": "^8.10.46",
"@types/node": "^12.12.32",
"aws-sdk": "^2.508.0",
"ts-node": "^8.10.2",
"typescript": "^3.8.3",
...
}
}
Here is my template.yaml:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
My APIs
Resources:
Profile:
Type: AWS::Serverless::Function
Properties:
CodeUri: Backend/Profile
Handler: dist/src/index.handler
Runtime: nodejs12.x
Events:
GetOneUser:
Type: Api
Properties:
...
...
This is an error I was facing with sam --version: 0.11.0 that disapeared when upgrading to sam --version: 0.52.0 using homebrew. I am currently moving CICD tool and I installed the same sam --version: 0.52.0 using pip3 as I cannot install homebrew on my CICD platform. So here it is:
- npm build run manually: works fine
- sam build (v 0.11.0) - Homebrew install:
ERR! code ENOLOCAL - sam build (v 0.52.0) - Homebrew install: works fine
- sam build (v 0.52.0) - pip3 install:
ERR! code ENOLOCAL
Steps to reproduce
cd __project_root__
npm install --prefix Backend/CommonT
npm run build --prefix Backend/CommonT
npm run test --prefix Backend/CommonT
npm install --prefix Backend/Profile
npm run build --prefix Backend/Profile
npm run test --prefix Backend/Profile
# I tried both with and without this line, it doesn't make a difference
# npm install -q --no-audit --no-save --production --unsafe-perm --prefix %ROOT%/CommonT
# This is the line that fails
sam build -t template.yaml --debug
Observed result
Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
'build' command is called
Collected default values for parameters: {}
8 resources found in the template
Found Serverless function with name='Profile' and CodeUri='Backend/Profile'
Found Serverless function with name='###Something else###' and CodeUri='###'
Collected default values for parameters: {}
Building function 'Profile'
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Found workflow 'NodejsNpmBuilder' to support capabilities 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Running workflow 'NodejsNpmBuilder'
Running NodejsNpmBuilder:NpmPack
NODEJS packaging file:/my/directory/job/Backend/Profile to /my/directory/temp/buildTmp/XXXXX
executing NPM: ['npm', 'pack', '-q', 'file:/my/directory/job/Backend/Profile']
NODEJS packed to canaree-profile-1.0.0.tgz
NODEJS extracting to /my/directory/temp/buildTmp/XXXXX/unpacked
NodejsNpmBuilder:NpmPack succeeded
Running NodejsNpmBuilder:CopyNpmrc
.npmrc copying in: /my/directory/temp/buildTmp/XXXXX/unpacked/package
NodejsNpmBuilder:CopyNpmrc succeeded
Running NodejsNpmBuilder:CopySource
NodejsNpmBuilder:CopySource succeeded
Running NodejsNpmBuilder:NpmInstall
NODEJS installing in: /my/directory/job/.aws-sam/build/Profile
executing NPM: ['npm', 'install', '-q', '--no-audit', '--no-save', '--production', '--unsafe-perm']
NodejsNpmBuilder:NpmInstall failed
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/aws_lambda_builders/workflows/nodejs_npm/actions.py", line 108, in execute
["install", "-q", "--no-audit", "--no-save", "--production", "--unsafe-perm"], cwd=self.artifacts_dir
File "/usr/local/lib/python3.6/dist-packages/aws_lambda_builders/workflows/nodejs_npm/npm.py", line 85, in run
raise NpmExecutionError(message=err.decode("utf8").strip())
aws_lambda_builders.workflows.nodejs_npm.npm.NpmExecutionError: NPM Failed: npm WARN deprecated [email protected]: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated [email protected]: https://github.com/lydell/resolve-url#deprecated
npm ERR! code ENOLOCAL
npm ERR! Could not install from "../CommonT" as it does not contain a package.json file.
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2020-06-22T12_03_11_067Z-debug.log
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/aws_lambda_builders/workflow.py", line 269, in run
action.execute()
File "/usr/local/lib/python3.6/dist-packages/aws_lambda_builders/workflows/nodejs_npm/actions.py", line 112, in execute
raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: NPM Failed: npm WARN deprecated [email protected]: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated [email protected]: https://github.com/lydell/resolve-url#deprecated
npm ERR! code ENOLOCAL
npm ERR! Could not install from "../CommonT" as it does not contain a package.json file.
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/XXXXXXXX-debug.log
Build Failed
npm ERR! /root/.npm/_logs/XXXXXXXX-debug.log
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 2034, 'exitReason': 'WorkflowFailedError', 'exitCode': 1, 'requestId': '42b13729-2136-4df8-8442-a25c0ae2ce19', 'installationId': 'acac1f1a-some-long-id-b854a9c82c0b', 'sessionId': 'fb6bba5f-some-other-long-id-ed535022cafb', 'executionEnvironment': 'CLI', 'pyversion': '3.6.9', 'samcliVersion': '0.52.0'}}]}
HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: NodejsNpmBuilder:NpmInstall - NPM Failed: npm WARN deprecated [email protected]: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated [email protected]: Please see https://github.com/lydell/urix#deprecated
npm ERR! code ENOLOCAL
npm ERR! Could not install from "../CommonT" as it does not contain a package.json file.
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/XXXXXXXX-debug.log
Expected result
When manually running the builds/tests with npm, everything compiles and run properly. I expect the same behavour to happen with sam build.
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
- OS:
Ubuntu 18.04.4 LTS -
sam --version: 0.52.0
@sriram-mv , sorry, I'm a bit new to github. Could you please explain what the added "type/feature" means? Does that mean this bug is being looked at? Thanks a lot
Here is a work around:
I did manage to "force" the creation of the CommonT package by adding a resource with the CommonT name in my template.yaml.
Here is my new file:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
My APIs
Resources:
CommonT: # <= This NEEDS to have the same name as the CommonT Typescript project!
Type: AWS::Serverless::Function
Properties:
CodeUri: Backend/CommonT
Handler: dist/src/index.handler
Runtime: nodejs12.x
Events:
GetCommonData:
Type: Api
Properties:
RestApiId: !Ref RestAPI
Path: /some/config/data/for/example
Method: get
Profile:
Type: AWS::Serverless::Function
Properties:
CodeUri: Backend/Profile
Handler: dist/src/index.handler
Runtime: nodejs12.x
Events:
GetOneUser:
Type: Api
Properties:
...
...
Then, when sam build creates the .aws-sam package, it starts by creating the folder CommonT for the resource with the same name. And it is then used as the common project as well.
It is dirty... but it allows to deploy for now.
This work around work only when the name of the directory is alphanumeric.
Here is my work around. It is not pretty but it works.
First, use npm link or yarn link on your local node modules.
Next, set the dependencies section in your lambda's package.json to a path that is relative to the Lambda Function Code Folder under .aws-sam.
My folders look like this:
|--.aws-sam
|----build
|------zipcodeFunction
|--------node_modules
|--local-modules
|----data-layer
|--src
|----zipcodes
So I start by going into the local-modules/data-layer directory and running yarn link.
Next, go into the src/zipcodes (my Lambda function) directory and run yarn link data-layer.
Now a sam build will work and all your development tooling works also because the yarn link in the lambda directory adds the file in node modules. When you do a sam build it will bundle the local node module for you as you intend.
Here's my work around. Simply put - copy the dependency project into the current root.
My project has three parts - frontend, backend (SAM) and "proj-common", common artifacts for both front and backend.
I first put the dependency in package.json "dependencies" as "file:../proj-commons", pointing to the local folder. This works for the IDE, but sam build fails. The reason is SAM build in a container. In that container the mapped dir cannot access proj-commons. Check .aws-sam/build dir for the files actually available in SAM build.
I tried to put "../proj-commons" in "files" of "package.json" and it does not work. I tried to symbol link "../proj-commons" to a folder under the root, and refer to the folder in both "dependencies" and "files", it does not work.
So the final work around is - in my build script, first copy "../proj-commons" into a directory under the SAM project root, and also put it inside "files" in "project.json".
Here's how my package.json looks like. Note the build script.
"dependencies": {
"aws-sdk": "^2.437.0",
"proj-commons": "file:./.dependency_by_copy/proj-commons"
},
"files": [
"built/",
".dependency_by_copy/"
],
....
"scripts": {
"build": ". ./env.sh; rm -fr ./.dependency_by_copy; mkdir .dependency_by_copy; cp -r ../proj-commons ./.dependency_by_copy; tsc; sam build; announce \"Project Built\"",
I am trying to add some shared code to a layer but I am getting this error as well.
The yarn link did not worked for me. But @bingtimren workaround did, thank you !
Before :
{
...
"scripts": {
"build": "yarn install"
},
"dependencies": {
"shared": "file:./../shared"
}
}
After :
{
...
"scripts": {
"build": "mkdir -p dist && rsync -avr ../../shared dist --exclude node_modules --exclude src && yarn install"
},
"dependencies": {
"shared": "file:./dist/shared"
},
"files": [
"dist"
]
}
You can find the full example here : https://github.com/fbn-roussel/aws-sam-typescript-webpack-backend/blob/main/layers/application-layer/package.json
Native support of local path dependency would be greatly appreciated :)
We are currently reviewing a PR that should address this issue: https://github.com/aws/aws-lambda-builders/pull/215
hi, are there any news reagrding that topic? we are highly interested in that beeing solved.
@cBiscuitSurprise thank you for the hard work building this critical functionality! 🙏
@mgrandis I and probably the entire AWS SAM community are wondering if there is any progress on this PR since Feb 26? are there any better workarounds we should be aware of? Is there any way we can help test this so it will move forward?
Thanks!
If you are willing to try CDK, NodejsFunction with esbuild works for me pretty well.
See
https://docs.aws.amazon.com/cdk/api/latest/docs/aws-lambda-nodejs-readme.html#local-bundling
https://docs.aws.amazon.com/cdk/api/latest/docs/aws-lambda-nodejs-readme.html#configuring-esbuild
We've been limping along using 8c33ad6 custom-installed into our build environments. As much as I'd love to see @mgrandis and I's work merged into the product, my team has been slowly making proper JS packages out of all the instances where we're using this feature.
I'm happy to continue helping get this across the finish line. On the other hand, we can live without it (once we're fully converted to proper JS packages).
Thanks, @bingtimren & @cBiscuitSurprise, for the replies! Sadly we are not using CDK for various reasons.
my team has been slowly making proper JS packages out of all the instances where we're using this feature.
With that, do you mean a package that is published to the npm repository? Unfortunately, our internal library is private and needs to be resolved by our build system and other dev machines. Is there a way to get around this?
My goal is to share a library between all of my lambda functions. I honestly don't have a preference on how it's happening.
@eladg Yep, we're slowly isolating our modules as needed into packages which we publish to our private npm package registry (we use GitLab's package registry -- it can be free, even for private registries) and then just installing them like anything else.
@cBiscuitSurprise that idea is quite the last option we should consider in our case. local modules are standard in npm so it should just work. the impact in introducing new registries for npm packages is quite big - credentials, build pipelines, local build etc. But at least it is a valid option.
what we have done so far, is to create a symlink for our local packages into the aws build dir. at least it is working, but still not a very good option, because there is no optimization within the package steps. and you have to put this peace of linking code to each function.
{
...
"scripts": {
"preinstall": "node -e \"const fs = require('fs'); const path = require('path'); if (process.env.sam_build && !fs.existsSync('../../lib')) { fs.symlinkSync((process.env.CODEBUILD_SRC_DIR || '..') +'/lib/', path.resolve('../../lib/'), 'junction')}\""
},
...
}
with that kind of workaround we can build and package using sam build on local machine and with cloudbuild.
In the end we need the support for local modules. :-)
Closing so that we can track in the above issue (Feature request: Build in source). There's multiple open issues relating to similar problems, so we want to have a central place to track.
⚠️COMMENT VISIBILITY WARNING⚠️
Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
⚠️COMMENT VISIBILITY WARNING⚠️
Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.
⚠️COMMENT VISIBILITY WARNING⚠️
Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.