[rush] Generate package.json as a build output

[octogonz]

This idea has been around for a while, but it seems we didn't have an issue tracking it.

Copied from https://github.com/microsoft/web-build-tools/issues/1503#issuecomment-527542375:

---

There are several problems that this could solve.

1. Multi-target builds

First, imagine that your publishing workflow produces three outputs, from the same source files:

• my-package/package-rush.json - the file that humans edit; this does not get published
• my-package/build/internal/package.json - this gets published to your company's private NPM registry, by running npm publish in the build/internal folder. This special release provides full access to APIs marked as @internal
• my-package/build/beta/package.json - published with a version number like [email protected]. The dependencies section will be rewritten to refer to beta versions. For example, if my-package/package-rush.json depends on [email protected] it will get rewritten as [email protected] when this file is generated by Rush
• my-package/build/public/package.json - the official release with @internal and @beta APIs trimmed in the .d.ts rollup

The idea would be that a multi-target rush build would produce all three outputs simultaneously, writing outputs into separate subfolders.

2. Reducing Git merge conflicts

When running rush version in a large monorepo, suppose that someone has edited a core library such as a Guid class, which nearly every project depends on. If there are 200 projects in the repo, rush version may need to increment the versions for 180 of these projects (since they all use the Guid class). This shows up as a huge Git diff that churns the dependencies section for 180 package.json files.

This often leads to annoying merge conflicts. For example, in their PR, a human might make an edit like this:

my-library/package.json (human PR)

  "dependencies": {
    "library-a": "1.2.3",
    "library-b": "1.2.3",
+    "library-c": "1.2.3",
    "library-d": "1.2.3"
  }

That PR will encounter a merge conflict, because in the master branch, the publishing bot made an edit like this:

my-library/package.json (version bump from bot)

  "dependencies": {
-    "library-a": "1.2.3",
+    "library-a": "1.2.4",
-    "library-b": "1.2.3",
+    "library-b": "1.2.4",
-    "library-d": "1.2.3"
+    "library-d": "1.2.4"
  }

These merge conflicts could be avoided by storing the lockstep version number in a data file like common/config/rush/version-policies.json. And then using a token like this:

my-library/package.json

  "dependencies": {
    "library-a": "%MY_SDK_VERSION%",
    "library-b": "%MY_SDK_VERSION%",
    "library-d": "%MY_SDK_VERSION%"
  }

During the build, when Rush is writing the output file my-package/build/public/package.json, it would substitute %MY_SDK_VERSION% with 1.2.3 (or 1.2.3-beta). The versions are still getting churned, but the churn is no longer in files managed by Git. Thus we avoid a merge conflict.

3. Simplifying the source files

One other advantage of this process is that my-package/package-rush.json could be treated more like a source file: It could have comments, which are forbidden in a standard NPM package.json. And maybe it could use a relaxed input format like JSON5 or YAML. Common boilerplate (e.g. fields like license and scripts) could be inherited from a toolchain template.

[octogonz]

In Gitter today, @renoirb brought up another potential application:

Reduce copy+pasting of common package.json contents throughout the monorepo.

For example, certain versions could be centrally specified, to simplify upgrades:

{
  . . .
  "dependencies": {
    "typescript": "%MY_TS_VERSION%",
    "tslib": "%MY_TSLIB_VERSION%"
  }
  . . .
}

...or maybe common boilerplate could be inherited from a toolchain template:

{
  "$extends": "my-toolchain/include/default-package.json",
  . . .
  "dependencies": {
    "some-lib": "1.2.3"
  }
  . . .
}

[renoirb]

Related, apart of managing versions of a dependencies between packages, but related to package.json maintenance. How about having a script to normalize fields, see nuxt/nuxt.js#6373

[renoirb]

As requested by @octogonz

With relation to normalizing package.json and/or simplifying maintenance in relation to dependencies.

---

Use-Case 1 Use source package-rush.json as a guide to configure generating package.json as build output

The idea of having placeholders of versions and replace them around. As package.json is a build artifact and no longer source. I see the benefit for managing merges.

Part of this is done in Nuxt.js' scripts/package.js, where each package has a sibling package.js to tell it what to do. It notably allow you to sync deoendencies, copy contributors, etc

• nuxt/nuxt.js scripts/package.js
• nuxt/nuxt.js scripts/package

And in each package: nuxt/nuxt.js distributions/nuxt-start/package.js

export default {
  build: true,
  hooks: {
    async 'build:done' (pkg) {
      const mono = pkg.load('../..')
      const nuxt = pkg.load('../nuxt')
      await pkg.copyFilesFrom(mono, [
        'LICENSE'
      ])
      pkg.copyFieldsFrom(nuxt, [
        'license',
        'repository',
        // ...
      ])
      await pkg.writePackage()
    }
  }
}

There is currently a PR in progress to factor out the files copying and field syncing part on nuxt/nuxt.js#6373. This might become an independent module other projects could import.

Use-Case 2 Slim down dependencies with meta pckages

Other issue is that we often want to slim down dependencies lists. If I have a meta package "@corp/conventions-typescript", "@corp/conventions-bundling-electron" , bext time somebody wants to make another distribution targeted to run inside Electron, he would only need to add the one dependency. Not many others. In npm land, ghost dependencies hides it. Nuxt.js leverages this very much.

Docs says to just add nuxt and lots of things are pulled in. They're ghost dependencies.

I guess my question here is more about where to read about, how to properly explicitly do meta packages consistently. Without hidden ghosts. Or without mingling between packages, and having a Monorepo root package.json

Maybe it's impossible and a solution is your first point. I can expand the use-cases later. I wanted to put this down to illustrate what I'm trying to solve. Nuxt.js scripts/package.js tool does

sync dependencies between all packages scripts/package.js.

One might want to make sure license, author, ... fields are all the same copyFieldsFrom, copy files around copyFilesFrom

All of this are package.json manipulation we might want to automate so we don't need to maintain them manually.

I like the idea of using a source package-rush.json as a guide to configure generating package.json as build output. They're two different use-cases, both equally useful.

[octogonz]

CC @MickeyPhoenix who was interested in specifying dependency versions in one place.

[octogonz]

If https://github.com/pnpm/pnpm/issues/3316 "workspace tags" are not implemented by the package manager, perhaps it could be applied by the proposed package-rush.json transform.

[dmichon-msft]

My concern about package.json being a generated file is that many editors have special auto-complete behavior for package.json in particular, and that other tools in the workspace that you could otherwise use prior to installing rush stop working. For example currently there are situations where I deliberately run npm install inside of a subfolder instead of fully invoking rush because I have a tiny tooling package I want to run in a pipeline. Admittedly the autoinstaller pattern is probably a viable alternative for this use case.

[dmichon-msft]

I'm now proposing https://github.com/pnpm/pnpm/issues/3816 as the preferred package manager support, since it also works to support alternate publishing conventions (e.g. having the authoritative local version specifier for a package live outside of package.json). Between the readPackage hook and the readPackageForPublish hook, Rush can freely extend the dependencies schema of package.json as it sees fit to support shared versions.

[dmichon-msft]

One opportunity of having package.json be a generated file is that it allows us to write different versions of package.json at build time vs. publish time, e.g. so that the build time dependencies list can contain only exact versions (for predictable monorepo dependency resolution), but the publish time dependencies list relaxes the specifiers to be more flexible for consumers. This also gives the opportunity to trim out any parts of package.json that are only relevant to development time (devDependencies; npm scripts other than install, postinstall, etc.; sections related to jest or other build tools).

We'll want to figure out how to have VSCode provide an intelligent editing experience for the dependencies section of the pakcage-rush.json (or whatever name) file, since the latest version autocomplete is nifty.

[daotoad]

Even having the ability to use jsonc, json5, or another file format that supports comments would be a boon.