Support mono-repos in deployment

[laurenzlong]

See: https://github.com/firebase/firebase-functions/issues/172

Version info

3.17.3

Steps to reproduce

Expected behavior

Actual behavior

[dinvlad]

This seems to work in my setup, i.e. deploy picks up packages from the root node_modules, even though package.json for that is located under the api/ workspace (I've used a different folder name instead of functions/). Is there anything else that needs fixing here?

EDIT: Moreover, I copy package.json into api/dist to be used by deploy.

// firebase.json
  ...
  "functions": {
    "source": "api/dist"
  },
  ...

So, 2 levels of nesting still resolve the root node_modules successfully.

[orouz]

@dinvlad could you share a repo?

[dinvlad]

@orouz unfortunately not yet, it's closed source for now.

[audkar]

Does anyone managed to tackle this problem? Sharing simple example project would be very useful.

[jthegedus]

@audkar Currently I just use lerna.js.org and it's run command to execute an npm script in each subfolder with this folder structure:

- service1/
|  - .firebaserc
|  - firebase.json
- service2/
|  - .firebaserc
|  - firebase.json
- app1/
|  - .firebaserc
|  - firebase.json
- app2/
|  - .firebaserc
|  - firebase.json
- firestore/
|  - firestore.rules
|  - firestore.indexes.json
- etc...

Ensuring the firebase.json files for each service don't stomp on one another is left up to the user. Simple conventions of using function groups and multi-site name targeting means this is solved for Cloud Functions and Hosting. Still haven't got a solution for Firestore/GCS rules yet, though splitting them up may not be ideal...

discussed here previously - https://github.com/firebase/firebase-tools/issues/1116

[audkar]

@jthegedus thank you for your reply. But I think issue of this ticket is different. I am trying to use yarn workspaces. And seems that firebase tools doesn't pickup symlink dependencies when uploading functions

[jthegedus]

Ah fair enough, I've avoided that rabbit hole myself

[dinvlad]

Could you elaborate what the issue is? As mentioned above, I just use bare Yarn with api and app workspaces in it, and I build them using yarn workspace api build && yarn workspace app build (with build script specific to each workspace). The build scripts

1. compile TS code with outDir into api/dist and app/dist respectively
2. copy the corresponding package.json files into dist directories
3. copy yarn.lock from the root folder, into dist directories

Then I just run yarn firebase deploy from the root folder, and it picks up both api/dist and app/dist without any hiccups. My firebase.json looks like

  "functions": {
    "source": "api/dist"
  },
  "hosting": {
    "public": "app/dist",

Unfortunately, I still can’t share the full code, but this setup is all that matters, afaik.

[dinvlad]

Also, I might be wrong but I think the firebase deploy script doesn’t actually use your node_modules directory. I think it just picks up the code, package.json, and yarn.lock from the dist directories, and does the rest.

[samtstern]

That's true. The default value of "functions.ignore" in firebase.json is ["node_modules"] so it's not uploaded. I believe you can override that though if you want to ship up some local modules.

On Mon, Jun 17, 2019, 6:58 PM Denis Loginov [email protected] wrote:

Also, I might be wrong but I think the firebase deploy script doesn’t actually use your node_modules directory. I think it just picks up the cod, package.json, and yarn.lock from the dist directories, and does the rest.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/firebase/firebase-tools/issues/653?email_source=notifications&email_token=ACATB2U73VS2KIILUVRFFB3P3A6NPA5CNFSM4EOR24GKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX46ABQ#issuecomment-502915078, or mute the thread https://github.com/notifications/unsubscribe-auth/ACATB2U3Q2TLVBICRJ3B5OLP3A6NPANCNFSM4EOR24GA .

[jthegedus]

@dinvlad Yes, it requires the package.json and whichever lock file you use as it installs deps in the cloud post deployment.

I believe the scenario originally outlined in the other issue was using a shared package within the workspace and some issues with scope-hoisting. As I was not using yarn this way I can only speculate from what I have read there.

[dinvlad]

@samtstern @jthegedus thanks, good to know!

[audkar]

Seems we all talk about different problems. I will try to describe yarn workspaces problem.

Problematic project

project layout

- utilities/
|  - package.json
- functions/
|  - package.json
- package.json

./package.json

{
  "private": true,
  "workspaces": ["functions", "utilities"]
}

functions/package.json

{
  <...>
  "dependencies": {
    "utilities": "1.0.0",
    <...>
  }
}

Problem

Error during function deployment:

Deployment error.
Build failed: {"error": {"canonicalCode": "INVALID_ARGUMENT", "errorMessage": "`gen_package_lock` had stderr output:\nnpm WARN deprecated [email protected]: use String.prototype.padStart()\nnpm ERR! code E404\nnpm ERR! 404 Not Found: [email protected]\n\nnpm ERR! A complete log of this run can be found in:\nnpm ERR!     /builder/home/.npm/_logs/2019-06-18T07_10_42_472Z-debug.log\n\nerror: `gen_package_lock` returned code: 1", "errorType": "InternalError", "errorId": "1971BEF9"}}

Functions works fine locally on emulator.

Solutions tried

Uploading node_modules (using functions.ignore in firebase.json). Result is same.

My guess that it is because utilities is created as syslink in node-modules node_modules/utilities -> ../../utilities

Could it be that firebase-tools doesn't include content of symlink'ed modules when uploading (no dereferencing)?

[dinvlad]

Sorry, could you clarify which folder your firebase.json lives in (and show its configuration section for functions)?

[audkar]

firebase.json was in root folder. Configuration was standard. Smth like this:

  "functions": {
    "predeploy": [
      "yarn --cwd \"$RESOURCE_DIR\" run lint",
      "yarn --cwd \"$RESOURCE_DIR\" run build"
    ],
    "source": "functions",
    "ignore": []
  },
  <...>

everything was deployed as expected (including node_modules) except node_modules/utilities which is symlink.

---

I manage to workaround this issue by writing few scripts which:

• create packages for each workspace (yarn pack). e.g. this creates utilities.tgz.
• moving all output to some specific dir.
• modifying package.json to use tgz files for workspace dependencies. e.g. dependencies { "utilities": "1.0.0" -> dependencies { "utilities": "file:./utilities.tgz"
• deploying that dir to firebase

output dir content before upload:

- dist
|  - lib
|  | -index.js
|  - utilities.tgz
|  - package.json <---------- This is modified to use *.tgz for workspaces

[0x80]

@audkar Today I ran into the same issue as you.

I am new to both Lerna and Yarn workspaces. As I understand it you can also just use Lerna. Would that help in any way?

Your workaround seems a bit complicated for me 🤔

Also wondering, what is `--cwd "$RESOURCE_DIR" for?

[audkar]

--cwd stands for "current working directory" and $RESOURCE_DIR holds value for source dir (functions in this case). Adding this flag will make yarn to be executed in functions dir instead of root

[0x80]

@audkar Ah I see. So you could do the same with yarn workspace functions lint and yarn workspace functions build

[0x80]

@dinvlad It is unclear to me why you are targeting the dist folder and copying things over there. If you build to dist, but leave the package.json where it is and point main to dist/index.js then things should work the same no? You should then set source to api instead of api/dist.

[0x80]

@dinvlad I learned the yarn workspace command from your comments, but can't seem to make it work for some reason. See here. Any idea?

Sorry for going a bit off-topic here. Maybe comment in SO, to minimize the noise.

[dinvlad]

@0x80 I copy package.jsonto api/dist and point firebase.json to api/dist so only the "built" files are packaged inside the cloud function. I'm not sure what will happen if I point firebase.json to api - perhaps it will still be smart enough to only package what's inside api/dist (based on main attribute in package.json). But I thought it was cleaner to just point to api/dist.

Re yarn workspace, I responded on SO ;)

[0x80]

@dinvlad it will bundle the root of what you point it to, but you can put everything that you don't want included in the firebase.json ignore list.

I've now used a similar workaround to @audkar.

{
  "functions": {
    "source": "packages/cloud-functions",
    "predeploy": ["./scripts/pre-deploy-cloud-functions"],
    "ignore": [
      "src",
	  "node_modules"
    ]
  }
}

Then the pre-deploy-cloud-functions script is:

#!/usr/bin/env bash

set -e

yarn workspace @gemini/common lint
yarn workspace @gemini/common build

cd packages/common
yarn pack --filename gemini-common.tgz
mv gemini-common.tgz ../cloud-functions/
cd -

cp yarn.lock packages/cloud-functions/

yarn workspace @gemini/cloud-functions lint
yarn workspace @gemini/cloud-functions build

And packages/cloud-functions has an extra gitignore file:

yarn.lock
*.tgz

[kaminskypavel]

here's what worked for me

- root/
|  - .firebaserc
|  - firebase.json
- packages/
  | - package1/
  | - functions/
    | - dist/
    | - src/
    | packages.json

and in the root/firebase.json :

{
	"functions": {
		"predeploy": "npm --prefix \"$RESOURCE_DIR\" run build",
		"source": "packages/functions"
	}
}

[0x80]

@kaminskypavel is your packages/functions depending on packages/package1 (or some other sibling package)?

[kaminskypavel]

@0x80 positive.

[0x80]

I think there was something fundamental I misunderstood about monorepos. I assumed you can share a package and deploy an app using that package without actually publishing the shared package to NPM.

It seems that this is not possible, because deployments like Firebase or Now.sh will usually upload the code and then in the cloud do an install and build. Am I correct?

@kaminskypavel I tried your approach and it works, but only after publishing my package to NPM first. Because in my case the package is private I initially got a "not found" error, so I also had to add my .npmrc file to the root of the cloud functions package as described here

@audkar Are you publishing your common package to NPM, or are you like me trying to deploy with shared code which is not published?

[StephenHaney]

@0x80 I'm with you on this understanding - I think Firebase Function deployments are just (erroneously) assuming that all packages named in package.json will be available on npm, in the name of speeding up deployments.

As yarn workspace setups are becoming more popular, I imagine more folks are going to be surprised that they can't use symlinked packages in Firebase Functions – especially since they work fine until you deploy.

[StephenHaney]

With npm adding support for workspaces, we have an ecosystem standard for how local packages should work.

Since this issue is over a year old, any update from the Firebase side on plans (or lack of plans) here?

I think it's a pretty cool opportunity – Firebase's array of services begs for a good monorepo setup.

[michelepatrassi]

+1 on this, cloud functions usually will need to share some common code (e.g. interfaces) with other apps and a nice way to deal with this is a monorepo (e.g. lerna) or using symlinks directly. I took the latter and solved by creating some scripts. The concept is quite easy: I copy what's needed inside the functions directory and I remove it after

Here's how I did it with this directory structure:

- root/
|  - .firebaserc
|  - firebase.json
| - ...
- functions/
 | - src/
 | - package.json
 | - pre-deploy.js
 | - post-deploy.js
 | - ....
- shared/
 | - src/
 | - package.json
 | - ....

content of pre-deploy.js

const fs = require("fs-extra");

const packageJsonPath = "./package.json";
const packageJson = require(packageJsonPath);

(async () => {
    await fs.remove(`./shared`);
    await fs.copy(`../shared`, `./shared`);

    packageJson.dependencies["@project/shared"] = "file:./shared";

    await fs.writeFile(packageJsonPath, JSON.stringify(packageJson, null, 2));
})();

content of post-deploy.js

const packageJsonPath = "./package.json";
const packageJson = require(packageJsonPath);
const fs = require("fs-extra");

(async () => {
    await fs.remove(`./shared`);

    packageJson.dependencies["@project/shared"] = "file:../shared";

    await fs.writeFile(packageJsonPath, JSON.stringify(packageJson, null, 2));
})();

Then update firebase.json like this (add the build script if you need, I build before in my pipeline)

  "functions": {
    "source": "functions",
    "predeploy": [
      "npm --prefix \"$RESOURCE_DIR\" run pre-deploy"
    ],
    "postdeploy": [
      "npm --prefix \"$RESOURCE_DIR\" run post-deploy"
    ]
  },

If you do the build, inside the dist or lib directory you should now have two siblings: functions and shared (this happened because of the shared dependency). Make sure to update the functions package.json main to point to lib/functions/src/index.js to make the deploy work.

For now it's solved but that's a workaround, not a solution. I think that firebase tools should really support symlinks

[Stradivario]

@michelepatrassi inspired by what you have remind me i created firelink library for managing this case. It uses internally rsync to copy recursive files.

https://github.com/rxdi/firelink

npm i -g @rxdi/firelink

Basic usage Assuming that you have monorepo approach and your packages are located 2 levels down from the current directory where package.json is located.

package.json

  "fireDependencies": {
    "@graphql/database": "../../packages/database",
    "@graphql/shared": "../../packages/shared",
    "@graphql/introspection": "../../packages/introspection"
  },

Executing firelink it will Copy packages related with folders then will map existing packages with local module install "@graphql/database": "file:./.packages/database", then will execute command firebase and will pass rest of the arguments from firelink command. Basically firelink is a replacement for firebase CLI since it spawns firebase at the end when finish his job copying packages and modifying package.json!

Regards!