zig icon indicating copy to clipboard operation
zig copied to clipboard

`std.Build.Step.Compile.create` with `Options.use_llvm = false` breaks fuzz testing

Open Paladynee opened this issue 10 months ago • 1 comments

Zig Version

0.15.0-dev.97+677b2d62e

Steps to Reproduce and Observed Behavior

similar to #20990 in nature.

$ zig version
0.15.0-dev.97+677b2d62e

Minimal repro:

  1. zig init
  2. in build.zig change exe_unit_tests:
const exe_unit_tests = b.addTest(.{
    .root_module = exe_mod,
    .use_llvm = false,
});
  1. zig build test --fuzz Observed behavior:
eurydice@serenity:~/coreign/minimal$ zig build test --fuzz
info: web interface listening at http://127.0.0.1:37931/
info: hint: pass --port 37931 to use this same port next time
thread 59931 panic: start index 1 is larger than end index 0
/snap/zig/13959/lib/std/Build/Fuzz/WebServer.zig:685:17: 0x159270e in addEntryPoint (build)
        for (pcs[1..], 1..) |elem_addr, i| {
                ^
/snap/zig/13959/lib/std/Build/Fuzz/WebServer.zig:566:56: 0x154148d in coverageRun (build)
            .entry_point => |entry_point| addEntryPoint(ws, entry_point.coverage_id, entry_point.addr) catch |err| switch (err) {
                                                       ^
/snap/zig/13959/lib/std/Thread.zig:510:13: 0x1507afd in callFn__anon_71977 (build)
            @call(.auto, f, args);
            ^
/snap/zig/13959/lib/std/Thread.zig:1403:30: 0x14c3ee1 in entryFn (build)
                return callFn(f, self.fn_args);
                             ^
/snap/zig/13959/lib/std/os/linux/x86_64.zig:126:5: 0x14bc6c1 in clone (build)
    asm volatile (
    ^
???:?:?: 0x0 in ??? (???)
error: the following build command crashed:
/home/eurydice/coreign/minimal/.zig-cache/o/fdcab04bb682fef78333aab175a3a144/build /snap/zig/13959/zig /snap/zig/13959/lib /home/eurydice/coreign/minimal /home/eurydice/coreign/minimal/.zig-cache /home/eurydice/.cache/zig --seed 0x6a556cf9 -Zbf398f7019233e9f test --fuzz

Expected Behavior

Expected to see normal fuzzer results:

eurydice@serenity:~/coreign/minimal$ zig build test --fuzz
info: web interface listening at http://127.0.0.1:39513/
info: hint: pass --port 39513 to use this same port next time
[0/1] Fuzzing
└─ main.test.fuzz example

Paladynee avatar Mar 31 '25 20:03 Paladynee

Updated with minimal reproduction.

Paladynee avatar Apr 06 '25 17:04 Paladynee

Note you do not actually need .use_llvm = false to trigger this. If you just run zig init then zig build test --fuzz it will give you the same error. I guess this is because use_llvm is set to false by default though because setting it to true oddly makes it work. I'm on 0.15.0-dev.767+201c0f54a for reference.

presentfactory avatar Jun 10 '25 06:06 presentfactory

The underlying issue here is that the self-hosted x86 backend does not provide the fuzzer the __sancov_cntrs and __sancov_pcs1 sections.

The missing __sancov_pcs1 section is the culprit for the apparent issue as the fuzzer gets zero for the start and end of the section making it seem there are 0 pcs (which cannot normally happen.) As shown in the stack trace, the web server assumes at least one 1 pc.

As @presentfactory pointed out, if you build with use_llvm = true, it will work since it does not use the self-hosted backend. This is also true for release modes; you should run fuzz testing on ReleaseSafe anyways for the performance benefits (and rerun on Debug with bad inputs.)

gooncreeper avatar Jun 23 '25 17:06 gooncreeper

As @presentfactory pointed out, if you build with use_llvm = true, it will work since it does not use the self-hosted backend. This is also true for release modes

This is happening with the llvm backend too.

I just tried this with 0.16.0-dev.319+4d1b15bd9 and i'm seeing the crash with debug and test_exe.use_llvm = true,. Same with a release safe build. Same when setting exe.use_llvm = true,

$ zig version
0.16.0-dev.319+4d1b15bd9
$ mkdir asdf && cd asdf
$ zig init
$ zig build test --fuzz --release=safe --webui=[::1]:45589
info(web_server): web interface listening at http://[::1]:45589/
thread 542504 panic: start index 1 is larger than end index 0
~/zig/stage3/lib/zig/std/Build/Fuzz.zig:428:17: 0x1455573 in addEntryPoint (std.zig)
        for (pcs[1..], 1..) |elem_addr, i| {
                ^
~/zig/stage3/lib/zig/std/Build/Fuzz.zig:313:56: 0x13d9355 in coverageRun (std.zig)
            .entry_point => |entry_point| addEntryPoint(fuzz, entry_point.coverage_id, entry_point.addr) catch |err| switch (err) {
                                                       ^
~/zig/stage3/lib/zig/std/Thread.zig:511:13: 0x134d120 in callFn__anon_82012 (std.zig)
            @call(.auto, f, args);
            ^
~/zig/stage3/lib/zig/std/Thread.zig:1383:30: 0x12c1bd8 in entryFn (std.zig)
                return callFn(f, self.fn_args);
                             ^
~/zig/stage3/lib/zig/std/os/linux/x86_64.zig:119:5: 0x1248975 in clone (std.zig)
    asm volatile (
    ^
???:?:?: 0x0 in ??? (???)
error: the following build command crashed:
.zig-cache/o/8d664209a4672f0195975a80ab4764c3/build ~/zig/stage3/bin/zig ~/zig/stage3/lib/zig ~/fuzzing-example/asdf .zig-cache /home/travis/.cache/zig --seed 0xbeeebfb8 -Zb102de79da5bd393 test --fuzz --release=safe --webui=[::1]:45589

This is a fresh local zig build. Same crash with zigup downloaded 0.16.0-dev.254+6dd0270a1.

travisstaloch avatar Sep 21 '25 01:09 travisstaloch

$ zig build test --fuzz --release=safe --webui=[::1]:45589

If you pass -Doptimize=ReleaseSafe instead of --release=safe with the example you gave it works fine.

For the other examples you listed, you will have to give more complete reproductions. Also you may want to pass use_llvm through std.Build.TestOptions instead of modifying it directly.

gooncreeper avatar Sep 21 '25 14:09 gooncreeper

If you pass -Doptimize=ReleaseSafe instead of --release=safe with the example you gave it works fine.

For the other examples you listed, you will have to give more complete reproductions. Also you may want to pass use_llvm through std.Build.TestOptions instead of modifying it directly.

Thanks for the response. I just tried again and things seem to be working fine. I guess there was some kind of issue on my end. I did notice afterward a bunch of orphaned test processes still running. Not sure how that happened. But maybe that was it. Anyway sorry for the noise.

Btw, --release=safe seems to work fine too.

travisstaloch avatar Sep 21 '25 18:09 travisstaloch

It's taken me a while to report, but I've noticed that the fuzz tester has different issues on ReleaseSafe since at least Zig 0.16.0-dev+452.

Here's a log from 2025-09-29 immediately following a zig init:

$ zig build test -Doptimize=ReleaseSafe --fuzz
info(web_server): web interface listening at http://[::1]:40913/
info(web_server): hint: pass '--webui=[::1]:40913' to use the same port next time
test
└─ run test failure
thread 64671 panic: input file 'in' is in use by another fuzzing process
/nix/store/bm3hi0584zz268a6fcc4dkjk5dyap390-zig-0.16.0-dev.452+1f7ee99b3/lib/fuzzer.zig:405:37: 0x10ba268 in fuzzer_init_test (fuzzer)
                error.WouldBlock => @panic("input file 'in' is in use by another fuzzing process"),
                                    ^
/nix/store/bm3hi0584zz268a6fcc4dkjk5dyap390-zig-0.16.0-dev.452+1f7ee99b3/lib/compiler/test_runner.zig:419:34: 0x107a2ca in test.fuzz example (test)
        fuzz_abi.fuzzer_init_test(&global.test_one, .fromSlice(builtin.test_functions[fuzz_test_index].name));
                                 ^
/nix/store/bm3hi0584zz268a6fcc4dkjk5dyap390-zig-0.16.0-dev.452+1f7ee99b3/lib/compiler/test_runner.zig:181:29: 0x107c2a5 in main (test)
                test_fn.func() catch |err| switch (err) {
                            ^
/nix/store/bm3hi0584zz268a6fcc4dkjk5dyap390-zig-0.16.0-dev.452+1f7ee99b3/lib/std/start.zig:618:22: 0x107b682 in posixCallMainAndExit (test)
            root.main();
                     ^
/nix/store/bm3hi0584zz268a6fcc4dkjk5dyap390-zig-0.16.0-dev.452+1f7ee99b3/lib/std/start.zig:232:5: 0x107b30d in _start (test)
    asm volatile (switch (native_arch) {
    ^
???:?:?: 0x0 in ??? (???)
error: the following command terminated unexpectedly:
./.zig-cache/o/e924fb9089fd0eafda6c1c45ab440a79/test --cache-dir=./.zig-cache --seed=0xc4e9f9bf --listen=-
[1/1] Fuzzing

sagehane avatar Oct 12 '25 09:10 sagehane