zig icon indicating copy to clipboard operation
zig copied to clipboard

add a fuzz test for zig fmt

Open gooncreeper opened this issue 9 months ago • 5 comments

This fuzz test checks several properties of zig fmt are upheld, namely idempotency, textual equivilence, and no trailing whitespace.

All functions in the fuzz test have @disableInstrumentation for performance and since their branches are not interesting to the fuzzer.

Closes #10418 Closes #14654 Closes #15929 Closes #18088 Closes #20108 Closes #22778 Closes #23071 Closes #23722 Closes #24065

gooncreeper avatar May 04 '25 19:05 gooncreeper

Very nice! You know what else might be a nice property to test is that running fmt a second time does not change the output.

Also, I think this test will be much more efficient once we have a smith. I don't know how far along @mlugg got on that, but that's a fun project. Basically take a fuzz input and convert it into a valid AST.

andrewrk avatar May 04 '25 19:05 andrewrk

My zig-smith has taken a back seat for now, but I also don't think it would be a great test of the parser. My goal with that project has been to emit programs which compile correctly and do interesting things in the context of incremental compilation, and which, crucially, are deterministic at runtime and do not trigger IB. Those are big restrictions, so the thing I was working on was unfortunately quite limited in terms of syntax. I would definitely be interested in something more generalized, but I'd have to figure out how to make it useful in the context of testing incremental compilation!

mlugg avatar May 04 '25 23:05 mlugg

don't think it would be a great test of the parser

Right but we're talking about testing the renderer. The renderer happens to also be a great way to test the parser, which is why they are related. But my point is that it would provide the ability to test the idempotency property of zig fmt. That's going to be more efficient with valid AST.

andrewrk avatar May 06 '25 17:05 andrewrk

Just updated this PR to also test idempotency and some other stuff, and fixed most of the bugs it found.

gooncreeper avatar Jul 29 '25 22:07 gooncreeper

Thanks for the update, I have this open in a tab (among many...)

andrewrk avatar Jul 29 '25 22:07 andrewrk