zig icon indicating copy to clipboard operation
zig copied to clipboard

zig build: run steps in parallel

Open andrewrk opened this issue 3 years ago • 0 comments

Finally, after this PR, when you run zig build, it will actually use all your cores to do stuff in parallel.

After fixing any issues that crop up in zig's own build script, this should reduce the CI time, as well as make it more palatable to use zig build for local testing. For example, zig build test-translate-c in this branch took 0.8s for me vs 12s for master.

Merge Checklist:

  • [x] make sure the .pdb file is not regressed
  • [x] #14744
  • [x] is the call to make() in CompileStep is problematic?
  • [x] eliminate calls to process.exit(1) in make() implementations, instead set the stderr of step result. this way we don't end up with dead, useless threads.
  • [x] avoid printing to stderr in make() implementations. capture it, and set it in step.result if the step fails. print it later after the step graph has been executed.
  • [x] actually it should print the stderr directly after it fails, just make sure to lock the stderr mutex and don't interrupt the progress bar output from the build runner
  • [x] give build_runner its own progress bar
  • [x] add a -j argument to override the default thread count
  • [x] zig's build script appears to have several issues when parallelism is introduced
  • [x] move the test-cases back over to the build system to take advantage of running things at the same time
  • [ ] accept a seed option to the build runner and shuffle the dependency order using it
  • [x] semantic error information communicated to the build runner
  • [x] progress information communicated to the build runner
  • [ ] introduce build_runner.phase enum {declare, make} and assert the phase in many functions
  • [ ] handle panics during make() gracefully. currently they dump stack trace to stderr, and the thread is lost, deadlock
  • [ ] make uninstall step do it based on inspecting install steps, rather than with "push installed file"
  • [ ] extract WriteSourceFileStep out from WriteFileStep
  • [ ] unify the RunStep checks code with CheckFileStep code
  • [ ] make progress better. instead of 1 line terminal progress, make it check the terminal width and use multiple lines
  • [ ] rename FooStep to Step.Foo
  • [ ] learn target information from a binary on disk and use that for RunStep rather than knowledge of the target ahead of time.
  • [ ] figure out why compiling a static library doesn't report as cached the second time
  • [ ] remove std.log usage from std.Build.Cache in favor of proper error reporting
  • [ ] remove output_dir
  • [ ] file bug report for miscompilation (related: #14783)
  • [ ] build runner: actually limit the peak RSS rather than enforcing it post-hoc
  • [ ] extract struct fields from std.Build which should go into a struct shared by all Build instances. maybe std.Build gets renamed to std.Package and std.Build becomes the root struct.
  • [ ] support G, M, GiB, MiB, GB, MB, etc. shorthands in --maxrss arg
  • [ ] add Windows support for detecting maxrss of child process
  • [x] warning(module): unexpected EOF reading cached ZIR for crypto/bcrypt.zig
  • [ ] make std.testing.expectEqualStrings more generic and then re-use it in RunStep
  • [ ] add .h file generation tests back
  • [ ] make the cwd of RunStep take a Cache.Directory
  • [ ] when the cwd needs to participate in the cache manifest of a RunStep
  • [ ] put an addCheckFile as a method on FileSource
  • [ ] add more kinds of checks - endswith, does not match
  • [ ] collect error return traces from errors other than error.MakeFailed
  • [ ] ld.lld: warning: Linking two modules of different target triples: '/home/andy/Downloads/zig/zig-cache/o/610cf441612621c1d4bea0844fb8daea/test.o' is 'arm-unknown-linux-musleabihf' whereas 'ld-temp.o' is 'armv6kz-unknown-linux-musleabihf' it looks like zig is failing to put the subarch into the llvm triple
  • [ ] remove the static_lib_as_sysem_lib behavior. that was never supposed to work that way.
  • [ ] port uuid link test over to the new system
  • [ ] add a skip reason to show up in the summary. useful one will be "foreign binary"
  • [ ] add the incremental compilation tests back
  • [ ] instead of embedding the compiler in the check-cases process, spawn the compiler as a sub-process and communicate the updates using the compiler protocol.
  • [ ] 'warning(link): directory not found' let's make this a hard error in the CLI. it is happening from config.h generated with outdated ZIG_CMAKE_PREFIX_PATH
  • [ ] add the C emit-h tests back
  • [ ] eliminate std.Build.CheckObjectStep.runAndCompare
  • [x] test-cases: make it not link libc in the case of backend llvm
  • [ ] custom test runner results in garbage printed to stderr
  • [x] set ZIG_DEBUG_COLOR depending on build runner ttyconf
  • [ ] rename --verbose to --verbose-child-process and redefine it to: print to stderr the commandline before spawning any child process. audit all uses of the verbose flag and all child process invocations.
  • [ ] take away EmitOption.emit_to option and make it give a FileSource
  • [ ] make TestResult communicate the error name
  • [ ] make two RunStep with side-effects not clobber each other

andrewrk avatar Feb 14 '23 20:02 andrewrk