Specifying -buildifier in buildozer causes "running buildifier: pipe: too many open files"

[artem-zinnatullin]

Simplified example:

buildozer -buildifier buildifier 'set kind my_java_library' '//somepkg/...:%java_library'

Where buildozer and buildifier are normal 0.29.0 binaries installed from Homebrew

It partially succeeds but fails on many files with:

/somepath/x/BUILD: running buildifier: pipe: too many open files
/somepath/y/BUILD: running buildifier: fork/exec /usr/local/bin/buildifier: too many open files
fixed /somepath/z/BUILD

Here are my kernel limits related to max open files (macOS 10.14.6):

$ sysctl kern.maxfiles
kern.maxfiles: 10485760

$ kern.maxfilesperproc
kern.maxfilesperproc: 1048576

Both are very high and far from number of BUILD files we have in the project.

Setting -P 1 doesn't fix the issue.

[artem-zinnatullin]

Running buildozer with built-in formatter works, but we need to set custom buildifier to provide custom buildifier options

[artem-zinnatullin]

As pointed by @styurin internally there is another file limit in macOS

Default on my machine:

$ launchctl limit maxfiles
	maxfiles    256            unlimited

Bumped it up:

$ sudo launchctl limit maxfiles 64000 4000000

$ launchctl limit maxfiles
	maxfiles    64000          4000000

But buildozer -buildifier buildifier still failed :(

[arturdryomov]

@artem-zinnatullin, please specify the approximate count of BUILD files. It is hard to reproduce this without recreating the hierarchy.

[arturdryomov]

First of all, @artem-zinnatullin is a LIAR. Just kidding. buildozer is not available in Homebrew (neither Core nor Bazel one). What I did:

$ go get github.com/bazelbuild/buildtools/buildozer 

And it indeed reproduces with this:

PACKAGES_DIR="bazel_generated_packages"

touch "WORKSPACE"

mkdir -p "${PACKAGES_DIR}"

for PACKAGE_INDEX in {1..100}; do
  mkdir -p "${PACKAGES_DIR}/${PACKAGE_INDEX}"
  echo "java_library()" > "${PACKAGES_DIR}/${PACKAGE_INDEX}/BUILD"
done

${HOME}/go/bin/buildozer -buildifier buildifier 'set kind my_java_library' "//${PACKAGES_DIR}/...:%java_library"

rm -rf "WORKSPACE"
rm -rf "${PACKAGES_DIR}"

[arturdryomov]

BTW, setting file descriptors count via ulimit -n 4096 works. Reverting it back to 256 produces errors above.

[pmbethe09]

by default NumIO = 200 -- you can probably work around this for now by setting -numio=20 or something similarly low.

The fix is presumably to throttle the calls to cmd.Run in the runBuildifier function to some smaller number of active calls.

[arturdryomov]

Seems like -numio 20 doesn’t help either (with or without the -P 20 argument). Neither does -numio 2.

This issue feels shaky to me from two fronts.

• I don’t see a good reason behind bundling Buildifier invocations in Buildozer ones. I mean, it is possible just to run Buildifier after Buildozer as a separate command.
• Running Buildozer without the Buildifier argument works fine on any amount of input files (I’ve tried up to 10 000). At the same time, I’ve tried to see open file descriptors using the macOS Activity Monitor and it seems like a single Buildozer process is spawning, then 8 or so Buildifier processes. The Buildozer process opens 256 descriptors and doesn’t free them. Feels like a file descriptor leak somewhere.