Allow importing handler from layers

[ewindisch]

It would be useful to be able to load handler modules from layers! (I do this with other runtimes)

Options would be to include /opt/nodejs/node_modules in the search path, or to allow absolute imports.

[mhart]

This should work already – are you having problems?

[ewindisch]

Correct. The import seems to look in the "LAMBDA_TASK_ROOT" for the handler.

$ serverless invoke -f hello -l
{
    "errorType": "Error",
    "errorMessage": "Cannot find module '/var/task//opt/nodejs/node_modules/@iopipe/iopipe'",
    "stackTrace": [
        "    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)",
        "    at Function.Module._load (internal/modules/cjs/loader.js:562:25)",
        "    at Module.require (internal/modules/cjs/loader.js:690:17)",
        "    at require (internal/modules/cjs/helpers.js:25:18)",
        "    at getHandler (/opt/bootstrap.js:148:15)",
        "    at start (/opt/bootstrap.js:25:15)",
        "    at Object.<anonymous> (/opt/bootstrap.js:20:1)",
        "    at Module._compile (internal/modules/cjs/loader.js:776:30)",
        "    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)",
        "    at Module.load (internal/modules/cjs/loader.js:653:32)"
    ]
}

[ewindisch]

same error if I use @iopipe/iopipe.handler

{
    "errorType": "Error",
    "errorMessage": "Cannot find module '/var/task/@iopipe/iopipe'",
    "stackTrace": [
        "    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)",
        "    at Function.Module._load (internal/modules/cjs/loader.js:562:25)",
        "    at Module.require (internal/modules/cjs/loader.js:690:17)",
        "    at require (internal/modules/cjs/helpers.js:25:18)",
        "    at getHandler (/opt/bootstrap.js:148:15)",
        "    at start (/opt/bootstrap.js:25:15)",
        "    at Object.<anonymous> (/opt/bootstrap.js:20:1)",
        "    at Module._compile (internal/modules/cjs/loader.js:776:30)",
        "    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)",
        "    at Module.load (internal/modules/cjs/loader.js:653:32)"
    ]
}

[mhart]

Oh sorry – I thought you were talking about NODE_PATH

Can you outline the use case again?

[ewindisch]

@mhart I have a layer that contains my handler. In my personal use-case, that handler is a wrapper that then does an import and load of another handler... but it's also completely valid for users to have their entire application placed within layers.

Example serverless.yml:

service: aws-nodejs-lambci

provider:
  name: aws
  runtime: provided
  layers:
    - arn:aws:lambda:us-east-1:553035198032:layer:nodejs10:14
    - arn:aws:lambda:us-east-1:146318645305:layer:IOpipeNodeJS810:9

functions:
  hello:
    handler: '@iopipe/iopipe.handler' #'/opt/nodejs/node_modules/@iopipe/iopipe.handler'
    environment:
      IOPIPE_HANDLER: handler.hello

[mhart]

Oh, this works with existing AWS runtimes? I thought the handler always tried to resolve from LAMBDA_TASK_ROOT

[ewindisch]

@mhart The official AWS ones do not for NodeJS, but this is confirmed to work with the official Java and Python runtimes from AWS . I've been nudging AWS to get this working in Node as well... but also, more immediately, seeing if we can get the popular custom Node runtimes to support this (it is already supported in NSolid by Nodesource).

[mhart]

Hmmm. I'm a little wary of diverging from AWS too much. I could see how absolute paths could be supported, but how do you distinguish between a module called index and a file called index.js?

Looking through the AWS Node.js 10 code, it looks like it might support ../../opt/nodejs/node_modules/@iopipe/iopipe.handler (but 8.10 and 6.10 won't) – I might consider modifying the logic to allow this too if that suits?

[ewindisch]

My testing has been unsuccessful in using path traversals via '..' to enable this /w the official runtime.

Lambda has always supported importing from node_modules, this would ideally just allow importing from /opt/nodejs/node_modules... using the absolute path would be a solution here to avoid unintended consequences; this is already a requirement of the NSolid runtime and would help reduce drift between runtimes.

Simply allowing any import via an absolute path (or restricted to the task root and paths beginning with /opt) would suffice for me, and wouldn't cause any breakages moving to this runtime.

[mhart]

Interesting, this works for me:

mkdir opt
cd opt
npm install @iopipe/iopipe
cd ..
docker run -v $PWD/opt:/opt lambci/lambda:nodejs10.x ../../opt/node_modules/@iopipe/iopipe.handler

[mhart]

I haven't updated lambci/lambda:nodejs10.x in a couple of weeks, they might've changed the code to disallow this? But I would assume that would work fine on the nodejs10.x runtime. It currently outputs:

{
  "errorType": "Error",
  "errorMessage": "No IOPIPE_HANDLER environment variable set."
}

[ewindisch]

I'll try that!

On Tue, Jun 11, 2019 at 5:13 PM Michael Hart [email protected] wrote:

Interesting, this works for me:

mkdir opt cd opt npm install @iopipe/iopipe cd .. docker run -v $PWD/opt:/opt lambci/lambda:nodejs10.x ../../opt/node_modules/@iopipe/iopipe.handler

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/lambci/node-custom-lambda/issues/20?email_source=notifications&email_token=AAAR354NSVELVEQTO2C7JE3P2AIQPA5CNFSM4HXDFL42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXOQT6Q#issuecomment-501025274, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAR356YRUJNLGLCJDHOSEDP2AIQPANCNFSM4HXDFL4Q .

-- -Erica

[ewindisch]

Playing with this more, it seems that the official nodejs10.x runtime now supports the absolute path, but with lambci, I get Cannot find module '/var/task//opt/nodejs/node_modules/@iopipe/iopipe. Using '../../' didn't work and if it did, it would differ from AWS which errors if I use the .. path traversal)

[mhart]

Oh interesting, maybe they’ve updated the nodejs10.x runtime code (finally). Will look into it, thanks for the heads up 👍

Sent from my iPhone

On Jun 17, 2019, at 8:13 AM, Erica Windisch [email protected] wrote:

Playing with this more, it seems that the official nodejs10.x runtime now supports the absolute path, but I get the Cannot find module '/var/task//opt/nodejs/node_modules/@iopipe/iopipe error with lambci (using '../../' doesn't help and would also differ from AWS which errors if I use the .. path traversal)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[mhart]

Oh wow, yeah they've changed a lot of things since the last time I checked. Mostly good! Although it's interesting they've removed the ability to do relative requires. They now throw Use absolute paths when specifying root directories in handler names if you try to to relative paths.

I'll update this runtime accordingly (and lambci/lambda:nodejs10.x)