Converting some PNG files to CRN produce a file with a different checksum on different platforms or even with different compilers

[illwieckz]

Converting the test/unvanquished_64.png image to CRN produces the same file with:

• amd64, i686 with SSE2, arm64, armhf architectures,
• Linux, Windows, macOS systems,
• GCC, Clang, MinGW, MAVC AppleClang compilers.

But the test/test-colormap1-alpha1.png produces a CRN file with a different checksum on every platform or even compilers! Just compiling crunch with GCC or Clang makes the output file different! The same is verified with the other test alpha PNG files.

[illwieckz]

The content of the files is totally different, even file size changes between compilers or build types (Release, Debug…).

But when I convert the CRN back to PNG, the PNG are equals byte-to-byte. So I guess the stored data is the same, but the way it is compressed differs greatly.

[illwieckz]

With an LTO release build, same amd64 machine, and this command:

./crunch -file ../test/test-colormap1-alpha1.png -out test.crn -lzmastats

GCC 13:

Texture successfully written in 0.048s
Texture successfully processed in 0.068s
Input texture: 256x256, Levels: 1, Faces: 1, Format: A8L8
Input pixels: 65536, Input file size: 432, Input bits/pixel: 0.053
Output texture: 256x256, Levels: 9, Faces: 1, Format: DXT5
Output pixels: 87381, Output file size: 4694, Output bits/pixel: 0.430
LZMA compressed output file size: 3501 bytes, 0.321 bits/pixel

Total time: 0.070s

Clang 16:

Texture successfully written in 0.042s
Texture successfully processed in 0.065s
Input texture: 256x256, Levels: 1, Faces: 1, Format: A8L8
Input pixels: 65536, Input file size: 432, Input bits/pixel: 0.053
Output texture: 256x256, Levels: 9, Faces: 1, Format: DXT5
Output pixels: 87381, Output file size: 4683, Output bits/pixel: 0.429
LZMA compressed output file size: 3493 bytes, 0.320 bits/pixel

Total time: 0.066s

Clang 18:

Texture successfully written in 0.039s
Texture successfully processed in 0.057s
Input texture: 256x256, Levels: 1, Faces: 1, Format: A8L8
Input pixels: 65536, Input file size: 432, Input bits/pixel: 0.053
Output texture: 256x256, Levels: 9, Faces: 1, Format: DXT5
Output pixels: 87381, Output file size: 4683, Output bits/pixel: 0.429
LZMA compressed output file size: 3493 bytes, 0.320 bits/pixel

Total time: 0.058s

file size	lzma size	file
4694	3501	build-gcc13/test.crn
4683	3493	build-clang16/test.crn
4683	3493	build-clang18/test.crn

md5sum	file
9821ef65b2889189135d7c15a002bb23	build-gcc13/test.crn
c23b493c382d434817b5db2f6356a88d	build-clang16/test.crn
c23b493c382d434817b5db2f6356a88d	build-clang18/test.crn

But then when I do:

./crunch -file test.crn -out test.png

The produced PNG files are exactly the same:

md5sum	file
80b183f8729459b27b1802428dd32d05	build-gcc13/test.png
80b183f8729459b27b1802428dd32d05	build-clang16/test.png
80b183f8729459b27b1802428dd32d05	build-clang18/test.png

Something that crn-to-png test doesn't check, is that if the mipmaps are the same.

[illwieckz]

If I transcode to DDS this way:

./crunch -file test.crn -out test.dds

The file sizes are the same:

file size	file
87536	build-gcc13/test.dds
87536	build-clang16/test.dds
87536	build-clang18/test.dds

But the checksums are not:

file size	file
77dd1eace6ebabced58cbe4b5ca89573	build-gcc13/ test.dds
1708dea114dd93159fa3707ed84a95c6	build-clang16/test.dds
1708dea114dd93159fa3707ed84a95c6	build-clang18/test.dds

[illwieckz]

Unlike the CRN files that are almost completely different, the DDS files are only different from a dozen of bytes.

[illwieckz]

Disabling fast math makes the files all the same.

[illwieckz]

GCC 13:

Texture successfully written in 0.043s
Texture successfully processed in 0.061s
Input texture: 256x256, Levels: 1, Faces: 1, Format: A8L8
Input pixels: 65536, Input file size: 432, Input bits/pixel: 0.053
Output texture: 256x256, Levels: 9, Faces: 1, Format: DXT5
Output pixels: 87381, Output file size: 4695, Output bits/pixel: 0.430
LZMA compressed output file size: 3546 bytes, 0.325 bits/pixel

Total time: 0.064s

Clang 16:

Texture successfully written in 0.043s
Texture successfully processed in 0.058s
Input texture: 256x256, Levels: 1, Faces: 1, Format: A8L8
Input pixels: 65536, Input file size: 432, Input bits/pixel: 0.053
Output texture: 256x256, Levels: 9, Faces: 1, Format: DXT5
Output pixels: 87381, Output file size: 4695, Output bits/pixel: 0.430
LZMA compressed output file size: 3546 bytes, 0.325 bits/pixel

Total time: 0.059s

Clang 18:

Texture successfully written in 0.043s
Texture successfully processed in 0.065s
Input texture: 256x256, Levels: 1, Faces: 1, Format: A8L8
Input pixels: 65536, Input file size: 432, Input bits/pixel: 0.053
Output texture: 256x256, Levels: 9, Faces: 1, Format: DXT5
Output pixels: 87381, Output file size: 4695, Output bits/pixel: 0.430
LZMA compressed output file size: 3546 bytes, 0.325 bits/pixel

Total time: 0.066s

file size	lzma size	file
4695	3546	build-gcc13/test.crn
4695	3546	build-clang16/test.crn
4695	3546	build-clang18/test.crn

md5sum	file
be1670cbb24a7bfe1321d07f512b60c4	build-gcc13/test.crn
be1670cbb24a7bfe1321d07f512b60c4	build-clang16/test.crn
be1670cbb24a7bfe1321d07f512b60c4	build-clang18/test.crn

[illwieckz]

Alongside disabling fast-math on all platforms, disabling x87 with -mfpmath=sse -msse is also required on i686 to produce reproductible CRN files.

[illwieckz]

All platform-specific causes of producing different checksums were tracked down and we know why.

There may be some remaining cause of non-reproducibility like the amount of threads when using fast math (???), but that's not within topic of that issue.