shelljs
ShellJSInternalError: ENOENT: no such file or directory, open '/tmp/shelljs_...'
Node version (or tell us if you're using electron or some other framework):
v14.8.0
ShellJS version (the most recent version/Github branch you see the bug on):
0.8.4
Operating system:
Ubuntu
Description of the bug:
ShellJSInternalError: ENOENT: no such file or directory, open '/tmp/shelljs_87bea80abecef2d76bf8'
at Object.openSync (node:fs:497:3)
at Object.readFileSync (node:fs:393:35)
at execSync (/root/rtag-deploy/node_modules/shelljs/src/exec.js:89:17)
at Object._exec (/root/rtag-deploy/node_modules/shelljs/src/exec.js:205:12)
at Object.exec (/root/rtag-deploy/node_modules/shelljs/src/common.js:335:23)
Example ShellJS command to reproduce the error:
shelljs_1["default"].exec("git clone " + gitUrl)
It's part of the tsc generated js. In the typescript source it's as follows:
import shell from "shelljs";
...
shell.exec(`git clone ${gitUrl}`);
As far as I'm aware, this should work fine. Does the node script have permission to create the /tmp/shelljs_87bea80abecef2d76bf8 folder? This is one of the first setup steps shell.exec() takes.
Yes, I am running exec inside an express http handler. It usually works the first time after the server starts but on subsequent requests I get this error. Restarting the server makes it work again (for the next request).
We keep running into this as well, have not pinpointed exactly how it happens, though it's usually happening when package.lock is changed, it might be related to importing a package that also has shelljs as a dependency but still trying to confirm that.
It sounds like it might be useful to see the output of npm ls shelljs in your project.
# npm ls shelljs
npm WARN read-shrinkwrap This version of npm is compatible with lockfileVersion@1, but package-lock.json was generated for lockfileVersion@2. I'll try to do my best with it!
[email protected] /root/rtag-deploy
└── [email protected]
Ah, well that rules out the hypothesis of other projects importing shelljs 😜
Maybe something is tampering with global state we rely on? One thing which comes to mind is you could check the return value of shell.tempdir() before and after the error happens. Did the tempdir value change? Exec relies on this to create the tempfiles:
https://github.com/shelljs/shelljs/blob/71085dba6527f86ff89ddbf62f07d8656190aa74/src/exec.js#L33
No, it looks like the tempDir stays the same across multiple invocations in the same process:
/tmp
ShellJSInternalError: ENOENT: no such file or directory, open '/tmp/shelljs_d1729e3889dbd01d1b62'
at Object.openSync (node:fs:497:3)
at Object.readFileSync (node:fs:393:35)
at execSync (/root/rtag-deploy/node_modules/shelljs/src/exec.js:89:17)
...
/tmp
ShellJSInternalError: ENOENT: no such file or directory, open '/tmp/shelljs_bb39c8b8ba9d8d739f40'
at Object.openSync (node:fs:497:3)
at Object.readFileSync (node:fs:393:35)
at execSync (/root/rtag-deploy/node_modules/shelljs/src/exec.js:89:17)
...
/tmp
ShellJSInternalError: ENOENT: no such file or directory, open '/tmp/shelljs_763abb2af915605d5a2c'
at Object.openSync (node:fs:497:3)
at Object.readFileSync (node:fs:393:35)
at execSync (/root/rtag-deploy/node_modules/shelljs/src/exec.js:89:17)
Is it expected that the filename changes across invocations?
In case helpful:
# ls -l /tmp | grep shelljs
-rw-r--r-- 1 root root 2956 Mar 20 04:12 shelljs_0ed6daad799fb291f9a4
-rw-r--r-- 1 root root 2982 Mar 16 05:59 shelljs_15e250a45125f925f582
-rw-r--r-- 1 root root 2956 Mar 20 04:12 shelljs_1743a0607de41bc43f1d
-rw-r--r-- 1 root root 2982 Mar 16 04:07 shelljs_1c7686769b3f232366eb
...
Looks like those folders are only writable by root user. Are you running your project as sudo? I don't recommend doing that, I'm just trying to understand why the permissions are set up that way.
If I remember correctly, it's not always easy to distinguish between "file doesn't exist" and "I don't have permission to access the file." So ENOENT: no such file or directory might just mean we tried to write into the file but the write operation failed.
By the way, what version of Ubuntu are you all using? Maybe there's a policy change in a recent Ubuntu version.
I didn't think it was a permissions issue because shelljs was successfully able to create those files in the first place. Again, the confusing thing going on here is that exec is working on the first invocation (the first time I hit my endpoint) but seems to fail in subsequent invocations (all in the same process).
When I get the ENOENT error, it appears to me that the file shelljs is trying to find simply does not exist (rather than exists but the process can't read it).
It would still help to know (1) which Ubuntu version and (2) are you running your app as root user.
For folks who can repro? Would you mind cloning the ShellJS GitHub project, running npm install, and then running our unit tests? I'm curious if this same bug will reproduce when running tests.
If that doesn't repro, could someone throw together a minimal repro project for me to debug on my machine?
Have a repro on a machine right now
Error [ShellJSInternalError]: ENOENT: no such file or directory, open '/tmp/shelljs_ad97862519c0d373bb5e'
at Object.openSync (fs.js:458:3)
at Object.openSync (pkg/prelude/bootstrap.js:505:32)
at Object.readFileSync (fs.js:360:35)
at Object.readFileSync (pkg/prelude/bootstrap.js:758:36)
at execSync (/snapshot/cbox/node_modules/shelljs/src/exec.js:89:17)
at Object._exec (/snapshot/cbox/node_modules/shelljs/src/exec.js:205:12)
at Object.exec (/snapshot/cbox/node_modules/shelljs/src/common.js:335:23)
at AmiManager.listAmis (/snapshot/cbox/dist/src/ami.js)
at AmiManager.findLatestAmi (/snapshot/cbox/dist/src/ami.js)
at RemoteManagerImpl.create (/snapshot/cbox/dist/src/remote.js)
at process.runNextTicks [as _tickCallback] (internal/process/task_queues.js:62:5)
at Function.runMain (pkg/prelude/bootstrap.js:1489:13)
at internal/main/run_main_module.js:17:47
at async RemoteHelper.managed (/snapshot/cbox/dist/src/remote_helper.js)
at async App.remote (/snapshot/cbox/dist/src/app.js)
at async App.run (/snapshot/cbox/dist/src/app.js) {
errno: -2,
syscall: 'open',
code: 'ENOENT',
path: '/tmp/shelljs_ad97862519c0d373bb5e'
}
circleci@8c372a5a2b8f:/mnt/ramdisk/connect-web/apps/boost$ ls /tmp/shelljs_ad97862519c0d373bb5e
ls: cannot access '/tmp/shelljs_ad97862519c0d373bb5e': No such file or directory
circleci@8c372a5a2b8f:/mnt/ramdisk/connect-web/apps/boost$ ls -lai /tmp
total 48
185523670 drwxrwxrwt 1 root root 4096 Mar 26 21:17 .
536919395 drwxr-xr-x 1 root root 132 Mar 26 20:16 ..
1665210605 -rw-r--r-- 1 circleci circleci 33 Mar 26 20:17 .bash_env-605e41109325c576c9a6b669-0-build
1654700780 -rw-r--r-- 1 circleci circleci 1 Mar 26 20:16 circle-agent-runner.pid
12686349 drwx------ 2 circleci circleci 27 Mar 26 20:16 circleci-384491186
1654679311 -rw-r--r-- 1 circleci circleci 8480 Mar 26 21:17 shelljs_500ff00781eb5f411916
1644694636 -rw-r--r-- 1 circleci circleci 8505 Mar 26 21:09 shelljs_9b569e8a21b48f8081ac
1711563321 -rw-r--r-- 1 circleci circleci 8504 Mar 26 21:04 shelljs_be900fa1b051ee6d6d0f
1644694680 -rw-r--r-- 1 circleci circleci 0 Mar 26 21:09 shelljs_c7fc3ba66cc2df37acc3
1644694681 -rw-r--r-- 1 circleci circleci 0 Mar 26 21:09 shelljs_cc18a4bb9a819609f140
1269909563 drwxr-xr-x 3 root root 32 Sep 10 2020 v8-compile-cache-0
1199841387 drwxr-xr-x 3 circleci circleci 32 Mar 26 20:29 v8-compile-cache-3434
634530316 drwxr-xr-x 2 circleci circleci 30 Mar 26 20:29 yarn--1616790597818-0.05390654746905055
1721028193 drwxr-xr-x 2 circleci circleci 30 Mar 26 20:46 yarn--1616791600521-0.886893574929688
1660299171 drwxr-xr-x 2 circleci circleci 30 Mar 26 20:49 yarn--1616791795408-0.37333341769007733
619780091 drwxr-xr-x 2 circleci circleci 30 Mar 26 20:50 yarn--1616791842442-0.7622079033250815
circleci@8c372a5a2b8f:/mnt/ramdisk/connect-web/apps/boost$ uname -a
Linux 8c372a5a2b8f 4.15.0-1092-aws #98-Ubuntu SMP Wed Jan 6 22:22:51 UTC 2021 x86_64 GNU/Linux
trying to run your test suite but cant seem to figure how it works, did yarn install (took ages) and have ava setup, but yarn test seems broken.
all tests fail with
circleci@8c372a5a2b8f:~/shelljs$ yarn test
yarn run v1.22.4
$ ava test/*.js
Uncaught exception in test/cat.js
/home/circleci/shelljs/test/cat.js:1
import fs from 'fs';
^^^^^^
SyntaxError: Cannot use import statement outside a module
✖ test/cat.js exited with a non-zero exit code: 1
We just use the standard npm workflow (i.e., npm test should work). I don't know how yarn test works, but it seems like it's doing something different.
That specific error message looks like the code isn't being transpiled. ava test framework is supposed to have a transpiler builtin so that es6 module syntax is supported. If the error message persists with npm test, that sounds like a bug for the ava folks to look into.
We encountered this issue as well. I've created a minimal repo for reproduction: https://github.com/politics-rewired/shelljs-repro. This is written in TS because that's what we use, but happens in plain JS as well.
The basic workflow is:
- Clone remote remote to temporary directory
-
cdinto repo path - Make some changes to repo
- Remove repo path using
shell.rm
The problem was fixed for us by cding up out of the repo into $TMPDIR before rming the repo path.
Hmm, I get a different error when I try similar code locally. I noticed that exec() relies on the pwd() function to get the current directory. However, when you rm() the current directory, pwd() throws this error:
Error [ShellJSInternalError]: ENOENT: no such file or directory, uv_cwd
at process.wrappedCwd [as cwd] (internal/bootstrap/switches/does_own_process_state.js:128:26)
at _pwd ([PATH REDACTED]/node_modules/shelljs/src/pwd.js:13:34)
at execSync ([PATH REDACTED]/node_modules/shelljs/src/exec.js:31:10)
at Object._exec ([PATH REDACTED]/node_modules/shelljs/src/exec.js:205:12)
at Object.exec ([PATH REDACTED]/node_modules/shelljs/src/common.js:335:23)
at Object.<anonymous> ([PATH REDACTED]/repro.js:9:7)
at Module._compile (internal/modules/cjs/loader.js:999:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
at Module.load (internal/modules/cjs/loader.js:863:32)
at Function.Module._load (internal/modules/cjs/loader.js:708:14) {
errno: -2,
code: 'ENOENT',
syscall: 'uv_cwd'
}
Also, I noticed that subsequent shell.cd() commands fail, even if I provide absolute paths which I know for sure exist. This actually seems quite similar to what issue #1028 was describing.
Any updates/workarounds on this issue? I'm running into it on my 12.22.1 Node environment running on AWS EC2. The command in specific I'm tryin to execute is shell.exec('sed -i '' 1d file.cs'v). Thanks!
For folks who can repro? Would you mind cloning the ShellJS GitHub project, running
npm install, and then running our unit tests? I'm curious if this same bug will reproduce when running tests.
I ran the tests against ad91197. All tests passed:
$ npm run test
> [email protected] test /home/pi/dev/git/shelljs
> ava test/*.js
⠦ which › basic usage
580 tests passed
6 tests skipped
> [email protected] posttest /home/pi/dev/git/shelljs
> npm run lint
> [email protected] lint /home/pi/dev/git/shelljs
> eslint .
Executing the following failed on my raspberrypi:
const shell = require('shelljs');
shell.env.NODE_OPTIONS = '--max_old_space_size=4096';
shell.exec('ls -al');
$ node index.js
<--- Last few GCs --->
[8259:0x3e7f5d8] 166 ms: Mark-sweep 1.0 (3.5) -> 1.0 (3.5) MB, 6.8 / 0.0 ms (average mu = 0.678, current mu = 0.122) allocation failure GC in old space requested
[8259:0x3e7f5d8] 171 ms: Mark-sweep 1.0 (3.5) -> 1.0 (2.5) MB, 3.3 / 0.0 ms (average mu = 0.593, current mu = 0.217) last resort GC in old space requested
[8259:0x3e7f5d8] 175 ms: Mark-sweep 1.0 (2.5) -> 0.9 (2.5) MB, 4.1 / 0.0 ms (average mu = 0.438, current mu = 0.010) last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
0: ExitFrame [pc: 0xee5d00]
1: StubFrame [pc: 0xe5e20c]
Security context: 0x58b4ea35 <JSObject>
2: new constructor(aka Module) [0x24e1e2b5] [internal/modules/cjs/loader.js:142] [bytecode=0x24e31231 offset=34](this=0x24e3956d <Module map = 0x4dba48ed>,0x24e2e485 <String[#68]: /home/pi/dev/git/shelljs-test/node_modules/shelljs/src/exec-child.js>,0x5ebc00dd <null>)
3: ConstructFrame [pc: 0xe585ac]
4: StubFrame [pc: 0xf548...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
/home/pi/dev/git/shelljs-test/node_modules/shelljs/src/common.js:399
throw e;
^
Error [ShellJSInternalError]: ENOENT: no such file or directory, open '/tmp/shelljs_88c795440bd0ebc44834'
at Object.openSync (fs.js:462:3)
at Object.readFileSync (fs.js:364:35)
at execSync (/home/pi/dev/git/shelljs-test/node_modules/shelljs/src/exec.js:89:17)
at Object._exec (/home/pi/dev/git/shelljs-test/node_modules/shelljs/src/exec.js:205:12)
at Object.exec (/home/pi/dev/git/shelljs-test/node_modules/shelljs/src/common.js:335:23)
at Object.<anonymous> (/home/pi/dev/git/shelljs-test/index.js:3:7)
at Module._compile (internal/modules/cjs/loader.js:999:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
at Module.load (internal/modules/cjs/loader.js:863:32)
at Function.Module._load (internal/modules/cjs/loader.js:708:14) {
errno: -2,
syscall: 'open',
code: 'ENOENT',
path: '/tmp/shelljs_88c795440bd0ebc44834'
}
It looked like it was some Node.js memory issue in my case, the reserved heap space was greater than the available. The Error [ShellJSInternalError]: ENOENT: no such file or directory, open was just a side-effect from shelljs#exec,
Some env info:
$ uname -a
Linux raspberrypi 4.19.118-v7l+ #1311 SMP Mon Apr 27 14:26:42 BST 2020 armv7l GNU/Linux
$ free
total used free shared buff/cache available
Mem: 3999744 1871080 559256 360888 1569408 1615736
Swap: 102396 256 102140
node -v && npm -v
v12.22.8
6.14.15
After specifying a valid heap size, I could run my module:
const shell = require('shelljs');
shell.env.NODE_OPTIONS = '--max_old_space_size=1024';
shell.exec('ls -al');
$ node index.js
total 28
drwxr-xr-x 3 pi pi 4096 Dec 27 11:46 .
drwxr-xr-x 6 pi pi 4096 Dec 27 11:56 ..
-rw-r--r-- 1 pi pi 109 Dec 27 12:41 index.js
drwxr-xr-x 22 pi pi 4096 Dec 27 11:46 node_modules
-rw-r--r-- 1 pi pi 274 Dec 27 11:46 package.json
-rw-r--r-- 1 pi pi 5314 Dec 27 11:46 package-lock.json
Maybe shelljs can show the real error and not ENOENT. Anyways, thank you for the great lib :heart:
Update: if I use any invalid heap size setting, exec fails with ENOENT.
index.js:
const shell = require('shelljs');
shell.env.NODE_OPTIONS = '--max_old_space_size=alma';
shell.exec('echo');
$ node index.js
Error: illegal value for flag --max_old_space_size=alma of type size_t
Try --help for options
/home/pi/.config/nvm/versions/node/v12.22.8/bin/node: bad option: --max_old_space_size=alma
/home/pi/dev/git/shelljs-test/node_modules/shelljs/src/common.js:399
throw e;
^
Error [ShellJSInternalError]: ENOENT: no such file or directory, open '/tmp/shelljs_cb392bd3b0c9bf1b07d9'
at Object.openSync (fs.js:462:3)
at Object.readFileSync (fs.js:364:35)
at execSync (/home/pi/dev/git/shelljs-test/node_modules/shelljs/src/exec.js:89:17)
at Object._exec (/home/pi/dev/git/shelljs-test/node_modules/shelljs/src/exec.js:205:12)
at Object.exec (/home/pi/dev/git/shelljs-test/node_modules/shelljs/src/common.js:335:23)
at Object.<anonymous> (/home/pi/dev/git/shelljs-test/index.js:3:7)
at Module._compile (internal/modules/cjs/loader.js:999:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
at Module.load (internal/modules/cjs/loader.js:863:32)
at Function.Module._load (internal/modules/cjs/loader.js:708:14) {
errno: -2,
syscall: 'open',
code: 'ENOENT',
path: '/tmp/shelljs_cb392bd3b0c9bf1b07d9'
}
Update: if I use any invalid heap size setting,
execfails withENOENT.
index.js:const shell = require('shelljs'); shell.env.NODE_OPTIONS = '--max_old_space_size=alma'; shell.exec('echo');
I tried this on Ubuntu (node v12.22.8 installed through nvm) but I get a different error:
$ node index.js
Error: illegal value for flag --max_old_space_size=alma of type size_t
Try --help for options
[PATH REDACTED].nvm/versions/node/v12.22.8/bin/node: bad option: --max_old_space_size=alma
I could try on a raspberry pi (raspbian/raspberry pi OS), but I'm curious why that would be different. Are other folks hitting this error on other systems?
With Node 16+ and [email protected], it works as expected. The hardware and the OS are the same as before. (See https://github.com/shelljs/shelljs/issues/1024#issuecomment-1001552543.) No more ENOENT error.
Heap size exceeds the available:
cat ./package.json | jq .dependencies.shelljs && node -v && npm -v && uname -a && cat ./index.js && node index.js
"^0.8.5"
v16.18.0
8.19.2
Linux pi 4.19.118-v7l+ #1311 SMP Mon Apr 27 14:26:42 BST 2020 armv7l GNU/Linux
const shell = require('shelljs');
shell.env.NODE_OPTIONS = '--max_old_space_size=4096';
shell.exec('ls -al');
<--- Last few GCs --->
[26980:0x560cac8] 67 ms: Mark-sweep 0.3 (2.8) -> 0.3 (2.8) MB, 1.8 / 0.0 ms (average mu = 0.251, current mu = 0.029) allocation failure GC in old space requested
[26980:0x560cac8] 69 ms: Mark-sweep (reduce) 0.3 (2.8) -> 0.3 (1.8) MB, 2.3 / 0.0 ms (average mu = 0.142, current mu = 0.023) last resort GC in old space requested
[26980:0x560cac8] 73 ms: Mark-sweep (reduce) 0.3 (1.8) -> 0.3 (1.8) MB, 4.0 / 0.0 ms (average mu = 0.061, current mu = 0.012) last resort GC in old space requested
<--- JS stacktrace --->
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
Invalid heap size:
cat ./package.json | jq .dependencies.shelljs && node -v && npm -v && uname -a && cat ./index.js && node index.js
"^0.8.5"
v16.18.0
8.19.2
Linux pi 4.19.118-v7l+ #1311 SMP Mon Apr 27 14:26:42 BST 2020 armv7l GNU/Linux
const shell = require('shelljs');
shell.env.NODE_OPTIONS = '--max_old_space_size=alma';
shell.exec('ls -al');
Error: illegal value for flag --max_old_space_size=alma of type size_t
Try --help for options
/home/pi/.config/nvm/versions/node/v16.18.0/bin/node: bad option: --max_old_space_size=alma
/home/pi/.config/nvm/versions/node/v16.18.0/bin/node: bad option: --harmony-import-assertions
Hi there! I've been following this thread, is there any recommended course of action for resolving these kinds of errors? I am experiencing a very similar kind of ShellJS ENOENT error as what has been reported in this issue for the Docusaurus project: https://github.com/facebook/docusaurus/issues/9646
