[BUG] `npm ci` erroneously installs optional OS-constrained transitive dependency through direct shrinkwrap dependency

[restjohn]

Is there an existing issue for this?

• [X] I have searched the existing issues

This issue exists in the latest npm version

• [X] I am using the latest npm

Current Behavior

Consider a package, lib1, with the following characteristics.

• has an optionalDependency, with an OS constraint, such as fsevents package.json

  { ... "os": [ "darwin" ] ... }
  

• published with a shrinkwrap package descriptor
• shrinkwrap was generated on the platform compatible with the OS-constrained dependency

Developer A creates another package, app1, which depends on lib1, and generates app1's package-lock.json with npm install also on the platform matching said OS constraint.

Developer B, OR a CI process, on a different platform from said OS constraint, runs npm ci to install app1's dependencies. npm ci produces an error like the following.

[.../app1> npm ci
npm error code EBADPLATFORM
npm error notsup Unsupported platform for [email protected]: wanted {"os":"darwin"} (current: {"os":"linux"})
npm error notsup Valid os:  darwin
npm error notsup Actual os: linux
npm error A complete log of this run can be found in: /root/.npm/_logs/2024-07-02T12_41_06_204Z-debug-0.log

Examining app1's package-lock.json reveals that npm does not include an "optional": true entry in the package-lock block for lib1's fsevents dependency.

Expected Behavior

npm ci should retain the optional nature of the platform-specific dependency and proceed with a successful clean install of app1's dependencies regardless of the dependencies.

Steps To Reproduce

1. On a linux platform, clone the demo repository: https://github.com/restjohn/issues.npm.shrinkwrap_optional_dep.
2. Use npm 10.8.1, latest at the time of this writing.
3. cd app1
4. npm ci
5. npm should produce an error as in the Current Behavior section above.

Please see the README in the demo repository for quite a bit more detail about the nuances of this behavior.

Also note that the demo repository package lib1.shrinkwrap references the fsevents package through a devDependency, and npm should not be attempting to install lib1.shrinkwrap devDependencies from the app1 package anyway.

Environment

• npm: 10.8.1
• Node.js: 20.15.0
• OS Name: Debian GNU/Linux 12 (bookworm)
• System Model Name: Node 20 Slim Docker image
• npm config:

; node bin location = /usr/local/bin/node
; node version = v20.15.0
; npm local prefix = /npm_transitive_os_dep/app1
; npm version = 10.8.1
; cwd = /npm_transitive_os_dep/app1
; HOME = /root
; Run `npm config ls -l` to show all defaults.

[restjohn]

Aside from the problem of forcing installation of an optional dependency, another cause of this bug is that npm is installing the dev dependencies from the shrinkwrapped package. Apparently this is quite a long-standing issue.

[restjohn]

Others are having issues with shrinkwrap as well: https://github.com/npm/cli/issues/4323.

[Skosche3]

This is causing same issue in project I am working. Would love to see this fixed.

[YodaDaCoda]

I've been experiencing this issue & have attempted to debug what's going on in the npm internals. There's a few semi-related issues (#6138 #4323 #2921) but this one is the most recent so I'm leaving my thoughts here.

I've tried to attack this from a variety of angles, comparing the ideal tree with the existing tree, trying to understand how arborist's build-ideal-tree.js and reify.js stuff all comes together, etc.

It seems that in processing the shrinkwrap, if a dep is determined to be extraneous then none of the other attributes are retained.

https://github.com/npm/cli/blob/9cb9d5030b1fdb83e3ffa45b7c8d2de80a657768/workspaces/arborist/lib/shrinkwrap.js#L267-L282

In my particular case, the problematic dependancy is optional=true,dev=true. I haven't yet locked down how it ends up marked extraneous - might be related to reset-dep-flags.js

I had success modifying one line where the packages in the tree are checked for buildability - being unfamiliar with the npm codebase, I'm not sure if this is the Most Correct way of fixing this. It does allow npm ci to work on @restjohn's app1 package though.

https://github.com/npm/cli/blob/9cb9d5030b1fdb83e3ffa45b7c8d2de80a657768/workspaces/arborist/lib/arborist/build-ideal-tree.js#L197

-      if (!node.optional) {
+      if (!node.optional && !node.extraneous) {

[litoj]

I am running into this rn and have no solution at hand. When I try to reuse node packages when building in docker I run npm ci --dry-run | grep 'up to date' to ensure the cache is valid. However, this always fails, because npm ci always says it added fsevents. And that is despite the fact that I cannot install fsevents in docker even if I wanted to - npm i fsevents fails with something about bad platform. How come there are no solutions for this?