RFC: Package Distributions

[darcyclarke]

• adds RFC for new distributions capabilities

See rendered RFC

[Jolg42]

Hi there! Really happy to see this RFC and can provide an example from how we currently achieve the distribution of binaries at Prisma

We support something we called binaryTargets, see https://www.prisma.io/docs/reference/api-reference/prisma-schema-reference#binarytargets-options The list looks like this

[
  'darwin',
  'darwin-arm64',
  'debian-openssl-1.0.x',
  'debian-openssl-1.1.x',
  'rhel-openssl-1.0.x',
  'rhel-openssl-1.1.x',
  'linux-arm64-openssl-1.1.x',
  'linux-arm64-openssl-1.0.x',
  'linux-arm-openssl-1.1.x',
  'linux-arm-openssl-1.0.x',
  'linux-musl',
  'linux-nixos',
  'windows',
  'freebsd11',
  'freebsd12',
  'openbsd',
  'netbsd',
  'arm',
]

• We build our binaries for each of these "targets"
• You can also build your own engine, for example
  • if you "forked" or want to build it yourself
  • if we don't build for your "target"

The download of the binaries is achieved on postinstall of the @prisma/engines package, which fetches the binaries from CloudFront/S3.

Here you can see there are quite a few requirements to replace this with Package Distributions

• platform (covered by RFC)
• arch (covered by RFC)
• openssl version (for example, rhel-openssl-1.0.x is needed for AWS lambda)
• Escape hatch to provide your own custom-built binaries (not sure how this is covered at the moment)

Happy to be part of a next meeting if you find this interesting 😃

[Jolg42]

I forgot to mention that we also need in some cases to install more than one "binaryTarget" locally.

Example: For AWS Lambda a zip archive needs to prepared with what you want to deploy. If you are on macOS and are deploying a Lambda like this you then require 2 "binaryTarget",

• darwin if you want to execute the code locally on macOS
• rhel-openssl-1.0.x so that it can be added in the zip archive.

[styfle]

There are two features we need for Next.js that optionalDependencies doesn't currently provide:

1. Support for libc detection: glibc vs musl. Today we need a custom install script to run detect-libc and install the appropriate binary. This is necessary for every Linux install.
   • Supported in Yarn 3 since https://github.com/yarnpkg/berry/pull/3981
   • See also #438
2. Support for WASM fallback. Today we need a custom install script to run and detect if the current platform fails to match known distributions with prebuilt binaries. This is necessary for CodeSandbox/StackBlitz/etc.

Hopefully distributions can solve these problems.

[Brooooooklyn]

Besides detecting engines and other conditions automatically, also we need a mechanism to install dependencies for the other platform for cross-compiling:

Refs:

• https://github.com/napi-rs/node-rs/issues/376
• https://github.com/coder/code-server/issues/4804
• https://github.com/npm/arborist/pull/231#issuecomment-982492107

yarn 1.x can use --ignore-platform flag to download ignore all conditions. yarn 2+ can use supportedArchitectures to download specified platforms dependencies. pnpm can use --force flag to download ignore all conditions.

[saquibkhan]

• can check with vercel and esbuild
• can we achieve the same results without this feature
• will this help to remove post install script
• rethink script lifecycle

[Brooooooklyn]

Feedback from users of next.js

• Installation size is too large because of the libc field is not supported: https://github.com/vercel/next.js/issues/42641
• Platform-specific optional dependencies not being included in package-lock.json when reinstalling with node_modules present

Feedback from users of NAPI-RS

Noway to install prebuild packages for another platform.

This is really important for Electron developers and Docker users.

For example, users need to build their Windows ia32 distribution Electron app on Windows x64 and they need to install prebuild packages for windows-ia32 platform on it. Some users want to installation packages on Linux x64 gnu host so they can reuse the cache on it, then add the installed node_modules into node:lts-alpine (Linux x64 musl) containers to build their production image. In cross platform testing CI, developers want to install packages on host platform because it’s fast and can be benefited from the cache on the host. Install the packages is extremely slow in the qemu enumerate. https://github.com/Brooooooklyn/snappy/blob/main/.github/workflows/CI.yaml#L428-L444

Yarn 3 is the only solution for this case at present: https://yarnpkg.com/configuration/yarnrc#supportedArchitectures.

There are some issues mentioned about it:

• https://github.com/napi-rs/node-rs/issues/376
• https://github.com/swc-project/swc/issues/2898
• https://github.com/coder/code-server/issues/4804

Install wasm fallback without postinstall script

Always including the wasm fallback package as dependency will increase the installation size. There is no way to install the wasm fallback package without postinstall script when all the platform specified optionalDependencies are not suitable for the host platform.

What we are doing for this case is to use postinstall script to detect if the native binaries are installed successfully, if not, we will install the wasm fallback package in the postinstall script.

~~Install the fallback package contains sources and provide a way to rebuild it~~

It could be solved by 0049-link-packages-to-source-and-build

https://github.com/napi-rs/napi-rs/issues/1474

Like wasm package, we also can’t contain the sources for prebuild binary in the package by default due to the installation size concern. But some developers who are working for Large international companies complain about dependencies on prebuild native binaries; because the compliance policy at their company wanted to audit the source code (and how can we prove that the downloaded binary was built from the code?). So we need a way to publish the sources & infra codes for the prebuild binaries, but not install for users by default.

[ljharb]

@saquibkhan the use case I find valuable here is if I could make the "default" distribution for my package omit tests, source files, docs, etc, but provide some kind of "full" distribution that includes all those things.

The beneficial install bandwith impact of applying this to my 300-400 packages alone would likely save npm millions of dollars a year, conservatively.

[nlf]

i've made some updates to this document including several unanswered questions as well as detailing a known risk and adding in some other alternatives

[JoshuaKGoldberg]

👋 @darcyclarke we can see that you closed this on Dec 1 but it's not clear why. https://github.com/npm/statusboard/issues/647 says to refer to this PR for information. Is there somewhere that says why this RFC was closed? Thanks!