Include hash of 'Trusted Build' from source in release artifacts

[yezr]

The RCTab TDP includes these instructions for creating and verifying a build from from source

Note, however, that two independent runs of the jlinkZip task will result in ZIP files with different hashes! Although the file contents in the two ZIP files will be identical, the timestamps of those files will not be. Those timestamps are included in the hash and thus, the hashes will not match. As a result, generating a hash for the ZIP file itself is only suitable for verifying that this build has not been modified.

And continues

It is, however, possible to demonstrate that two builds are equivalent by stripping the timestamps.

The process for generating a single timestamp-independent hash signature for a build ZIP requires a few steps: not only does the ZIP file store timestamps for the files it contains, but one of the files within the ZIP file, lib/modules, is, in turn, a collection of time stamped files. Fully decomposing the build to its individual constituent files thus requires using the jimage utility (part of the Java 17 development kit) to extract the files contained within lib/modules.

• First, unzip the ZIP you built locally (by following the precise instructions in the “Full build” section above). For the rest of this example we’ll assume that you extracted the zip files to c:\rctab_v1.3.2_windows\
• Navigate to the top-level rcv directory (for our example c:\rctab_v1.3.2_windows\rcv\).
  • *Make sure not to execute any other commands that create, modify, or delete files within this directory (like opening the application). Any modifications to these files will change the result of the hashing process.
• Create a hash.bat file in the c:\rctab_v1.3.2_windows\rcv\ folder with the following command echo.>hash.bat
• Open hash.bat in notepad and paste in the text below these bullets (starting with ::hash.bat)
• Run the batch file in the cmd prompt by typing hash.bat then enter. It can take 30-45 minutes to complete as it is computing a hash for every single file

:: hash.bat
@echo off

:: NOTE: This script must be placed one level up from the rcv directory

echo Initiating batch hash procedure...
echo %date% %time%

setlocal EnableExtensions EnableDelayedExpansion

set "HASHFILE=all_hashes.txt"
set "TEMPHASHFILE=all_hashes_temp.txt"
set "EXTRACTIONDIR=.\rcv\modules_extracted"
set "MODULESFILE=.\rcv\lib\modules"

if exist %HASHFILE% (
	echo Deleting existing hash file, %HASHFILE% ...
    del %HASHFILE%
)

if exist %EXTRACTIONDIR% (
	echo Deleting existing extracted modules directory, %EXTRACTIONDIR% ...
    rmdir /s /q %EXTRACTIONDIR%
)

echo Extracting contents of modules file...
mkdir %EXTRACTIONDIR%
cd %EXTRACTIONDIR%
jimage extract ..\..\%MODULESFILE%

echo Temporarily relocating modules file...
cd ..\..
copy %MODULESFILE% .
del %MODULESFILE%

:: Calculate the hash for every file here and in all subdirectories, appending to the file (format "(filename) = (hash)")
echo Calculating hashes...
for /r .\rcv %%f in (*) do (
    <NUL set /p ="%%f = " >> %HASHFILE%
    C:\Windows\System32\certutil.exe -hashfile "%%f" SHA512 | findstr /v ":" >> %HASHFILE%
    echo hashing %%f
)

echo Restoring modules file...
move .\modules %MODULESFILE%

:: Replace the absolute paths to each file with relative paths (e.g. C:\temp\rcv => .\rcv)

echo Replacing absolute paths with relative paths in hash file...
set "SEARCHTEXT=%cd%"
set "REPLACETEXT=."
for /f "delims=" %%A in ('type "%HASHFILE%"') do (
    set "string=%%A"
    set "modified=!string:%SEARCHTEXT%=%REPLACETEXT%!"
    echo !modified!>>"%TEMPHASHFILE%"
)
del "%HASHFILE%"
rename "%TEMPHASHFILE%" "%HASHFILE%"

echo Sorting the hash file...
sort "%HASHFILE%" > "%TEMPHASHFILE%"
del "%HASHFILE%"
rename "%TEMPHASHFILE%" "%HASHFILE%"

echo Calculating the hash of the entire sorted hash file...
C:\Windows\System32\certutil.exe -hashfile %HASHFILE% SHA512

endlocal

Example output (Windows):

SHA512 hash of all_hashes.txt:
a3bc3f93a213f3542ca64d085a701174c154044a37e5c33c39adcbbc325e38c609ef1ae6f678bd34b4a477a85201d48dfec8b8b341b4c62b40f127bbfdadbb81
CertUtil: -hashfile command completed successfully.

Currently, a new release does not contain this source-built hash to verify against. Instead, it contains a hash of each of the six build artifacts: one installer and one .zip for each of the three OSes we build for.

Can we add an additional step in the release process to calculate the source-built hash and include it as an additional release artifact.

[yezr]

~~I will note here that I built the .zip on a Windows machine using the gradlew.bat jlinkZip command from the same source about 10 minutes apart. The SHA512 hash of the .zip from those two builds were identical, which was a surprise to me based on the comments above. The SHA512 was different from the windows .zip SHA512 in the release artifacts.~~

I spent some time this morning trying to recreate this. I am now getting different hashes from different builds as expected. I must have done something wrong the first time, please disregard!

I was also able to confirm that running the hash.bat code above correctly results in a matching hash for two different builds from the same source! yeehaw! 🤠

[tarheel]

I will note here that I built the .zip on a Windows machine using the gradlew.bat jlinkZip command from the same source about 10 minutes apart. The SHA512 hash of the .zip from those two builds were identical, which was a surprise to me based on the comments above. The SHA512 was different from the windows .zip SHA512 in the release artifacts.

Hmm, yeah, that's odd. Not sure how that's possible. Maybe the second run used cached data or something?

[artoonie]

FWIW, I tried to get this process to work, but it does not: https://github.com/beryx/badass-jlink-plugin/issues/113

Just noting here for posterity.

[yezr]

Removing this from 1.3.2 because we can add it to the release manually for 1.3.2. We can automate it later.

[yezr]

I've edited the wiki with steps on golden hash generation.

FWIW the steps assume you have the correct Java installed and are on an internet-connected computer. Is it possible to put the correct version of Java and Gradle into the root of the source folder so that there is no confusion on which is necessary for golden hash? Java versions are ~180mb and Gradle is ~120mb. I know it's a little overkill (current .zip of the source is ~60mb) but, just want to throw it out there.

[artoonie]

It seems reasonable to link them from the Wiki or from the README rather than including them directly?