[bug] EMFILE: too many open files

[konradpabjan]

What happened?

Stumbled on a failure that happened with v4. Apparently this doesn't happen with v3

Example run can be found here: https://github.com/spacetelescope/jdaviz/actions/runs/7266578163/job/19798598137

What did you expect to happen?

Successful artifact upload

How can we reproduce it?

Not sure, but I think the key here is a large number of files. In the logs it shows 8824 files which is quite a bit. Maybe some read stream isn't being closed somewhere 🤔

Anything else we need to know?

Nope

What version of the action are you using?

v4

What are your runner environments?

windows

Are you on GitHub Enterprise Server? If so, what version?

No

[zaikunzhang]

Also seen in my workflow.

Run actions/[email protected]
With the provided path, there will be [7](https://github.com/primalib/prima/actions/runs/7338125515/job/19983281711#step:4:8)3166 files uploaded
Artifact name is valid!
Root directory input is valid!
Beginning upload of artifact content to blob storage
node:events:492
      throw er; // Unhandled 'error' event
      ^

Error: EMFILE: too many open files, open '/home/runner/work/prima/prima/prima_prima_quadruple.1_20.ubln.single.231227_1211_start/HS63'
Emitted 'error' event on ReadStream instance at:
    at emitErrorNT (node:internal/streams/destroy:151:[8](https://github.com/primalib/prima/actions/runs/7338125515/job/19983281711#step:4:9))
    at emitErrorCloseNT (node:internal/streams/destroy:[11](https://github.com/primalib/prima/actions/runs/7338125515/job/19983281711#step:4:12)6:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -24,
  code: 'EMFILE',
  syscall: 'open',
  path: '/home/runner/work/prima/prima/prima_prima_quadruple.1_20.ubln.single.23[12](https://github.com/primalib/prima/actions/runs/7338125515/job/19983281711#step:4:13)27_1211_start/HS63'
}

Node.js v[20](https://github.com/primalib/prima/actions/runs/7338125515/job/19983281711#step:4:21).8.1

[robherley]

We can possibly fix with the file method in archiver:

• https://www.archiverjs.com/docs/archiver#file

Appends a file given its filepath using a lazystream wrapper to prevent issues with open file limits.

Right now we are just using the append operation and creating a read stream per file:

• https://github.com/actions/toolkit/blob/5430c5d84832076372990c7c27f900878ff66dc9/packages/artifact/src/internal/upload/zip.ts#L47-L49

[SMoraisAnsys]

Bumping as I'm facing the same issue on a repo :)

[sungaila]

Same issue here when I tried to upload 9,834 files for a single artifact. The compression-level made no difference.

Had to downgrade to upload-artifact@v3 and download-artifact@v3.

[qwerttvv]

actions/upload-artifact@main
  with:
    name: MSYS
    path: C:\MSYS
    if-no-files-found: warn
    compression-level: 6
    overwrite: false
  env:
    MSYSTEM: MINGW64
With the provided path, there will be 51574 files uploaded
Artifact name is valid!
Root directory input is valid!
Beginning upload of artifact content to blob storage
node:events:492
      throw er; // Unhandled 'error' event
      ^

Error: EMFILE: too many open files, open 'C:\MSYS\msys64\mingw64\lib\python3.11\distutils\tests\__pycache__\test_cygwinccompiler.cpython-311.pyc'
Emitted 'error' event on ReadStream instance at:
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -4066,
  code: 'EMFILE',
  syscall: 'open',
  path: 'C:\\MSYS\\msys64\\mingw64\\lib\\python3.11\\distutils\\tests\\__pycache__\\test_cygwinccompiler.cpython-311.pyc'
}

Node.js v20.8.1

same bug at latest https://github.com/actions/upload-artifact/tree/3a8048248f2f288c271830f8ecf2a1c5d8eb0e9a

actions/upload-artifact@v3 is good but warning

Warning: There are over 10,000 files in this artifact, consider creating an archive before upload to improve the upload performance.

[zaikunzhang]

Is this fixed yet?

[ehuelsmann]

Running into this with Linux too: https://github.com/ehuelsmann/LedgerSMB/actions/runs/8121634001/job/22200227067#step:6:26

[pcfreak30]

Ditto, just ran into the same problem :/

[rmunn]

I've switched https://github.com/actions/toolkit/pull/1723 (proposed fix for retaining file permissions in uploaded .zip files) to use zip.file(file.sourcePath) instead of zip.append(createReadStream(file.sourcePath)) so that that PR can also fix this bug at the same time. It needs a review from repo maintainers before it can go anywhere, though.

[socketsguru]

Is there a command that we can run in the docker container that allow us to increment the number of open file descriptors to 10k lets say for example??

Wouldn't that fix this issue?

[xanather]

Just found got this issue. Using v3 I get warnings its going to be killed soon aswell:

https://github.blog/changelog/2024-04-16-deprecation-notice-v3-of-the-artifact-actions/

This needs to be fixed before that happens @konradpabjan @robherley otherwise it will make it not possible to upload artifacts that contain over the OS file descriptor limit.

[reko-lygg]

My workaround is to compress first to a .tgz, then upload it. A clunky approach, but seems to do the trick for me. However, I suppose this might be a non-solution for some.

oneJob: 
  steps:
    - name: 🏗 Checkout branch
      uses: actions/checkout@v4
    # ...
    - name: 🗜️ Compress
      run: tar -czf ./upload.tar.gz ./build
    - name: ⬆️ Upload results
      uses: actions/upload-artifact@v4
      with:
        name: build-${{ github.event.number }}
        path: ./upload.tar.gz
        overwrite: true
        if-no-files-found: error
        retention-days: 1 # No need to keep these longer for now, until we can figure if there's a way to share these between jobs
        compression-level: 0 # We are already gzipping above, makes no sense to compress again. Remove wehen upload-artifact fixes its stuff

twoJob:
  needs: oneJob
  steps:
    - name: 🏗 Checkout branch
      uses: actions/checkout@v4
    - name: ⬇️ Download results
      uses: actions/download-artifact@v4
      with:
        name: build-${{ github.event.number }}
    - name: 🤯 Expand
      run: tar -xzf ./upload.tar.gz

[rmunn]

@konradpabjan - https://github.com/actions/toolkit/pull/1723 is a PR that should fix this bug (along with https://github.com/actions/upload-artifact/issues/38); it only needs a review from a GitHub team member before it can be merged. Would you have time to take a look at it? It's been nearly two months since I opened the PR, and I'd really like it to not sit there unreviewed for another two months.

[xanather]

Please merge this before v3 support is removed from GitHub cloud :(

[lczech]

And another bump from me to increase urgency :-)

[xanather]

I pay for GitHub enterprise which is quite expensive as a startup. The fact there is file handler leak in v4 on Windows for this long while v3 is about to be killed in November is unacceptable especially since GitHub provides Windows hosted cloud runners themselves.

Code which has fixed the issue has been presented that can either be merged or based on to get this fixed easily.

@robherley, @konradpabjan, @bethanyj28, @jtamsut please prioritize as November is not too far off.

[rmunn]

@lczech @xanather - I opened https://github.com/orgs/community/discussions/127202 a while ago to try to get some attention. Nobody from the GitHub team seems to have noticed it. If you could perhaps upvote it, and/or bump it with a comment, that might (MIGHT) help.

[rmunn]

A GitHub team member recommended in https://github.com/actions/toolkit/pull/1723#issuecomment-2243758561 that I split the fix for this bug out of https://github.com/actions/toolkit/pull/1723 which was also fixing the Unix file permissions issue. So I've created https://github.com/actions/toolkit/pull/1771 which just has the "too many open files" bugfix.

[robherley]

👋 Hey y'all, I've merged @rmunn's changes into toolkit.

Before we publish a new version, I've made a release candidate branch with the changes so that folks can try it out:

• https://github.com/actions/upload-artifact/compare/tmp-4.3.5-rc0

Feel free to test with:

-uses: actions/upload-artifact@v4
+uses: actions/[email protected]

[robherley]

The changes are now released in v4.3.5 (and v4) 🎉

• https://github.com/actions/upload-artifact/releases/tag/v4.3.5

[robherley]

Unfortunately the change for .file change we introduced a regression for symlinks.

I'm reopening this issue for now to track, but for folks running into this problem, please pin to v4.3.5!

The latest v4 release has reverted the changes to prevent regression.

[xanather]

Will this be re-released soon?