[Bug?]: yarn dlx -p not working with package.json scripts

[uchar]

Self-service

• [ ] I'd be willing to implement a fix

Describe the bug

If I use yarn dlx -p [email protected] build with this package.json scripts section

...
"scripts": {
  "build": "tsc"
}
...

I get this error :

Internal Error: Binary not found (build) for root-workspace-0b6124@workspace:.

What I'm trying to do is to find a replacement for yarn global add from yarn v1 without installing all dev dependencies in order to build a project

To reproduce

clone this repository https://github.com/uchar/yarnbugs/tree/yarn_dlx_error , move to branch yarn_dlx_error

Then run

docker build -t yarn_bug .

if you use RUN yarn dlx -p [email protected] tsc instead of RUN yarn dlx -p [email protected] build it starts working.

Environment

System:
    OS: Linux 5.15 Alpine Linux
    CPU: (16) x64 AMD Ryzen 7 5800H with Radeon Graphics
    Binaries:
    Node: 20.17.0 - /tmp/xfs-19d7b165/node
    Yarn: 4.4.1 - /tmp/xfs-19d7b165/yarn
    npm: 10.8.2 - /usr/local/bin/npm

Additional context

No response

[clemyan]

yarn dlx is specifically for running one of the bins of the downloaded packages. The main use case being running commands provided by npm packages when you don't have a project. (e.g. create-x packages)

What I'm trying to do is to find a replacement for yarn global add from yarn v1 without installing all dev dependencies in order to build a project

To quote yarn dlx's docs:

Using yarn dlx as a replacement of yarn add isn't recommended, as it makes your project non-deterministic (Yarn doesn't keep track of the packages installed through dlx - neither their name, nor their version).

By using yarn dlx as a replacement of yarn add, you

• have no control over the version of transitive dependencies used
• download more stuff over multiple runs because transitive dependencies are not locked
• open yourself to supply chain attacks because the tool used is not locked/checksummed
• are unable to work offline

When you are in a project, simply yarn add-ing a dependency is the recommended approach by a large margin.

[uchar]

@clemyan Hi, it's a common practice to create multi-stage Dockerfiles, where in one stage, we download all dependencies, and in the next, remove those not needed for production. e.g If you simply use yarn add, it will include the entire TypeScript package, which isn't necessary for the production stage and can result in a significantly larger final image. We definitely don't want a 2GB production Docker image just because TypeScript was included in the final build. While this might save some bandwidth during the build process, it leads to a bigger waste when people have to repeatedly download a large image.

My usage isnt replacement for yarn add, but more of a substitute for the old yarn add --global functionality.

For example, in Yarn 1, it was common to install packages like Nest.js and the TypeScript CLI solely for the build process. Not doing so with the new Yarn setup can result in unnecessarily large images .

FROM node:18-alpine AS dependencies
WORKDIR /app
COPY package.json ./
COPY yarn.lock ./
#install all production packages
RUN yarn  install --production --frozen-lockfile && yarn cache clean

FROM node:18-alpine3.16 AS builder
WORKDIR /app
COPY --from=dependencies /app/node_modules ./node_modules
#install nest.js cli just for build stage
RUN yarn global add @nestjs/[email protected] typescript
RUN yarn build

FROM node:18-alpine3.16 AS runner
#copy cached depenedencies again
#final image now have no typescript and nest.js cli included
COPY --from=dependencies /app/node_modules ./node_modules

[clemyan]

Hi, it's a common practice to create multi-stage Dockerfiles, where in one stage, we download all dependencies, and in the next, remove those not needed for production. e.g If you simply use yarn add, it will include the entire TypeScript package, which isn't necessary for the production stage and can result in a significantly larger final image. We definitely don't want a 2GB production Docker image just because TypeScript was included in the final build. While this might save some bandwidth during the build process, it leads to a bigger waste when people have to repeatedly download a large image.

We have yarn install --production (v1) and yarn workspaces focus --production (v4) for that, which you are already using.

For example, in Yarn 1, it was common to install packages like Nest.js and the TypeScript CLI solely for the build process.

Again, for this kind of packages, installing globally is problematic because of non-reproducible builds ("works on my machine").

Ever since npx was introduced, there are very few use cases for npm i -g that isn't better covered by either npm i -D or npx.

If a package is used in a project by multiple developers or across multiple environments (including across host/container), then it almost certainly belongs in devDependencies. If a package only generates/modifies project files that can be shared across developers/environments and the package itself is not required to utilize them (e.g. create-x, @yarnpkg/sdk), then npx/yarn dlx may be a better fit.

[uchar]

@clemyan Term "devDependencies" has a clear meaning , these are "developement" dependencies , and clearly it's not called "buildDependencies" . There is no reason for someone to install eslint,prettier,jest and tens of other plugin to just "build" a project. IMO usecase is very clear , One e.g install typescript ,nest.js or anyother CLI required for a build and then build a project without any other devDepencies , and in case of muli stage docker file final image will be very small because build stuffs will be only installed and use in build stage ( and since you mentioned security , using less package means more security too) If you look at the repo I send https://github.com/uchar/yarnbugs/blob/yarn_dlx_error/Dockerfile , the command RUN yarn dlx -p [email protected] build should work but it doesnt , not sure what we are discussing here It's a valid usecase and valid problem , these are serious problems IMO , I like how fast new yarn is but our team cant use it with this issues, I just migrate everything to npm again

[clemyan]

There is no reason for someone to install eslint,prettier,jest and tens of other plugin to just "build" a project

Sounds like you want a new kind of "buildDependencies" between production and dev?

using less package means more security too

Using less packages means more security, but just because it is not in your package.json doesn't mean you are not using it. Using something (e.g. via dlx) without saying you are using it is less security.

in case of muli stage docker file final image will be very small

The final image only contains production dependencies either way, so that's moot.

not sure what we are discussing here

Yes, we are comparing a lot of thing/usecases/options here. Let's establish some baselines here:

1. We are talking about usecases where package(s) are need to build the project but not to run the project. That's stuff like esbuild or nest build or tsc, not something like create-react-app or nest generate or @yarnpkg/sdks vscode
2. The options we are comparing are

• Global install (yarn global add (v1), npm i -g)
• Temporary install (yarn dlx, npx), this includes a hypothetic dlx that works for running scripts (e.g. yarn dlx -p typescript build)
• Dev-dependency install (yarn add dev, npm i -D)
• A hypothetical buildDependencies

Here are where each option stands across various metrics:

• Reproducibility - Whether the package and dependencies are locked and checksummed (also relates to security because reproducibility mitigates supply-chain attacks)
  • Global: No :x:
  • Temporary: No :x:
  • Dev-dependency: Yes :heavy_check_mark:
  • Build-dependency: Yes :heavy_check_mark:
• Security
  • Global: Low :x:
    • :x: Not auditable
    • :large_orange_diamond: Deprecations checked on initial install only
  • Temporary: Medium :large_orange_diamond:
    • :x: (Yarn Modern) Not audited
    • :heavy_check_mark: Deprecations checked per-run
  • Dev-dependency: High :heavy_check_mark:
    • :heavy_check_mark: Auditable
    • :heavy_check_mark: Deprecations checked per-install
  • Build-dependency: High :heavy_check_mark:
    • :heavy_check_mark: Auditable
    • :heavy_check_mark: Deprecations checked per-install
• Flexibity - fixing problematic transitive dependencies (via resolutions/overrides or patches)
  • Global: Low :x: - Not possible
  • Temporary: Low :x: - Not possible
  • Dev-dependency: High :heavy_check_mark: - Possible
  • Build-dependency: High :heavy_check_mark: - Possible
• Flexibity - using different version for different projects
  • Global: Low :x:
    • :x: Compatibility of globally-installed version with the project is not checked
    • :x: Must manually switch versions each time
    • :x: Peer dependencies cannot be resolved correctly across global/local packages
  • Temporary: High :heavy_check_mark:
    • :heavy_check_mark: Can specify version per-run
  • Dev-dependency: High :heavy_check_mark:
    • :heavy_check_mark: Can specify version per-workspace
  • Build-dependency: High :heavy_check_mark:
    • :heavy_check_mark: Can specify version per-workspace
• Bandwidth usage on build, assuming proper caching
  • Global: Low :heavy_check_mark:
    • :heavy_check_mark: Each needed package downloaded once over all builds (until updated)
  • Temporary: High :x:
    • :heavy_check_mark: Only download needed packages
    • :x: New transitive dependencies downloaded each run
  • Dev-dependency: High :x:
    • :heavy_check_mark: Each needed package downloaded once over all builds (until updated)
    • :x: Extraneous packages downloaded once over all builds (until updated)
  • Build-dependency: Low :heavy_check_mark:
    • :heavy_check_mark: Each needed package downloaded once over all builds (until updated)
    • :heavy_check_mark: Only download needed packages
• Final image size
  • Same for all, because only production dependencies are included

And, for a nice summary:

	Global	Temporary	Dev-dependency	Build-dependency (hypothetical)
Reproducibility	Low :x:	Low :x:	High :heavy_check_mark:	High :heavy_check_mark:
Security	Low :x:	Medium :large_orange_diamond:	High :heavy_check_mark:	High :heavy_check_mark:
Flexibity on fixing problematic transitive dependencies	Low :x:	Low :x:	High :heavy_check_mark:	High :heavy_check_mark:
Flexibity on using different version for different projects	Low :x:	High :heavy_check_mark:	High :heavy_check_mark:	High :heavy_check_mark:
Bandwidth usage on build	Low :heavy_check_mark:	High :x:	High :x:	Low :heavy_check_mark:
Final image size	Low :heavy_check_mark:	Low :heavy_check_mark:	Low :heavy_check_mark:	Low :heavy_check_mark:

---

Given that

• global and temporary installs significantly reduce reproducibility and security
• buildDependencies are only useful if one has separate build and dev/test environments, which AFAICT is pretty rare

I don't think any of those three would fit in Yarn's core offering. That said, there are a few options you can consider:

Just use devDependencies

Of course the hypothetical "buildDependencies" is the best at every metric since it is hypothesized precisely for this usecase, but "devDependencies" still blows every other option out of the water. The only thing "devDependencies" loses to "buildDependencies" is bandwidth usage on build. But, if you are caching the packages, then every package version is only downloaded once across all builds.

"Manually" move dependencies

You can write a small script that moves dev-dependencies into dependencies. That way you can control, per-environment, what deps you need:

$ ./move-deps.sh typescript @types/node # move typescript and @types/node from devDependencies to dependencies
$ yarn workspaces focus --production

Workspaces

Restructure your project so that each task (e.g. build, test, lint) is self-contained in a workspace, then you can use yarn workspaces focus to choose dependencies:

$ yarn workspaces focus --production @myapp/app @myapp/builder
$ cd ./packages/builder
$ yarn run build ../app

Yarn plugins

I said buildDependencies probably wouldn't fit in Yarn's core because it is too niche. Luckily, that's exactly why we have a plugin API. You can probably build a command that supports buildDependencies yourself using the source code for yarn workspaces focus as a reference. (Though you probably need to parse the package.json again instead of using workspace.manifest)

Heck, you can build a dlx variant that can run scripts yourself or even yarn global add using plugins.

[yarnbot]

Hi! 👋

It seems like this issue as been marked as probably resolved, or missing important information blocking its progression. As a result, it'll be closed in a few days unless a maintainer explicitly vouches for it.

[uchar]

@clemyan Hi there , sry for late response I was in hospital

You can write a small script that moves dev-dependencies into dependencies. That way you can control, per-environment, what deps you need:

I think you need to add another very important column which is "developer experience" or "ease of use" . surely we can write all sort of plugins to handle this but then instead of developing the core project one need to constantly develop this new stuffs and every new member of team need to understand these hacks that no one else knows

Flexibity - using different version for different projects Global: Low ❌ ❌ Compatibility of globally-installed version with the project is not checked ❌ Must manually switch versions each time ❌ Peer dependencies cannot be resolved correctly across global/local packages Temporary: High ✔️ ✔️ Can specify version per-run Dev-dependency: High ✔️ ✔️ Can specify version per-workspace Build-dependency: High ✔️ ✔️ Can specify version per-workspace

This is not valid comparision in case of Docker image , one install a package in a single stage of docker file and use it on that same stage, there won't be any compatibility issue here , Sure on a dev machine with tens of project it can happen but in case of a proper CI/CD pipeline it shouldnt cause any problem . I think main issue here is that you are trying to enforce some sort of opionated pattern and in return it removes a lot of flexibiilty for developers

Security Global: Low ❌ ❌ Not auditable 🔶 Deprecations checked on initial install only Temporary: Medium 🔶 ❌ (Yarn Modern) Not audited ✔️ Deprecations checked per-run Dev-dependency: High ✔️ ✔️ Auditable ✔️ Deprecations checked per-install Build-dependency: High ✔️ ✔️ Auditable ✔️ Deprecations checked per-install

Yes if you compare them like that it's true but my main point about security was that there is absolutely no point to install dev depencies in production or build stage of Docker , there might be all sort of package in dev dependencie each depends on thousands of other packages , yes yarn audit checks for knowns security issue but there is always chance something slips away . less package = less chance of an unsecure package so dev-dependency is defentily not more secure , it's e.g comparing installing only typescript for build or installing 30 different package in devDependencies each depends on couple of hundred more packages, which will have more chance to be secure ?

Bandwidth usage on build, assuming proper caching Global: Low ✔️ ✔️ Each needed package downloaded once over all builds (until updated) Temporary: High ❌ ✔️ Only download needed packages ❌ New transitive dependencies downloaded each run Dev-dependency: High ❌ ✔️ Each needed package downloaded once over all builds (until updated) ❌ Extraneous packages downloaded once over all builds (until updated) Build-dependency: Low ✔️ ✔️ Each needed package downloaded once over all builds (until updated) ✔️ Only download needed packages

Well all sort of problems starts here "assuming proper caching" , If we assume it then yes but out of the box it's not that obvious how one should do "proper caching" , I think there is lack of documentation about best practices e.g in my case I want to make an efficient Docker image not sure how exactly should I do it , this is not the case with old yarn ,

Reproducibility - Whether the package and dependencies are locked and checksummed (also relates to security because reproducibility mitigates supply-chain attacks) Global: No ❌ Temporary: No ❌ Dev-dependency: Yes ✔️ Build-dependency: Yes ✔️

Yes reproducibility comparision for dev and build is correct but for the other two it's not , one will e.g install typescript v5.0.2 , it's reproducible and will work , nothing will be changed in typescript v5.0.2 in different machines so both Global and Temporary will be also reproducible ( if one install global package with exact same version )

[yarnbot]

Hi! 👋

It seems like this issue as been marked as probably resolved, or missing important information blocking its progression. As a result, it'll be closed in a few days unless a maintainer explicitly vouches for it.