Future proposal: support for test report generation

[flotter]

How about we add additional functionality in spread to support test report generation with task level command support to add metrics and checks results ?

So, in addition to MATCH at the task level, perhaps we can have something like:

REPORT_CHECK_PASS <category> <name> REPORT_CHECK_SKIPPED <category> <name>

REPORT_METRIC <name> <units> <value>

My implementation currently has the report generator written in Python, and the report is posted as a comment with each PR.

It would be great to also support generating a report artifact, such as a PDF (which should be easy from the markdown/HTML/Latex used below - which is supported by GH for rendering)

My spread currently generates this:

Test Report

Checks and metrics in this report was produced by running the tests on x86 / noble inside an architecture specific QEMU emulator, without KVM acceleration. Metrics related to size are accurate, but performance metrics related to time based measurements are effectively benchmarking the performance of the host machine, not the emulated platform.

Coverage

Category	Check	Base	Platform	Suite	Task	Baseline	Actual
Machine boot	First boot after flashing	24.04	qemu-amd64	smoke	test-boot	Pass	Pass
			qemu-arm64			Pass	Pass
			qemu-riscv64			Pass	Pass
	Reboot request from client		qemu-amd64			Pass	Pass
			qemu-arm64			Pass	Pass
			qemu-riscv64			Pass	Pass
Machine refresh	Refresh to next revision		qemu-amd64		test-refresh	Pass	Pass
			qemu-arm64			Pass	Pass
			qemu-riscv64			Pass	Pass

Metrics

Machine boot to Networking/NTP

Base	Platform	Suite	Task	Units	Baseline	Actual	Delta
24.04	qemu-amd64	smoke	test-boot	milliseconds	30000	32712	${\color{orange}\verb|9.04%|}$
	qemu-arm64				30000	45793	${\color{red}\verb|52.64%|}$
	qemu-riscv64				30000	28214	-5.95%

Machine boot to PID1

Base	Platform	Suite	Task	Units	Baseline	Actual	Delta
24.04	qemu-amd64	smoke	test-boot	milliseconds	20000	26822	${\color{red}\verb|34.11%|}$
	qemu-arm64				20000	27843	${\color{red}\verb|39.22%|}$
	qemu-riscv64				20000	22214	${\color{orange}\verb|11.07%|}$

Machine bootloader time

Base	Platform	Suite	Task	Units	Baseline	Actual	Delta
24.04	qemu-amd64	smoke	test-boot	milliseconds	10000	18992	${\color{red}\verb|89.92%|}$
	qemu-arm64				10000	23173	${\color{red}\verb|131.73%|}$
	qemu-riscv64				10000	17634	${\color{red}\verb|76.34%|}$

Machine reboot (only) time

Base	Platform	Suite	Task	Units	Baseline	Actual	Delta
24.04	qemu-amd64	smoke	test-refresh	milliseconds	30000	30798	2.66%
	qemu-arm64				30000	54696	${\color{red}\verb|82.32%|}$
	qemu-riscv64				30000	30260	0.87%

Machine reboot (with refresh) time

Base	Platform	Suite	Task	Units	Baseline	Actual	Delta
24.04	qemu-amd64	smoke	test-refresh	milliseconds	30000	38535	${\color{red}\verb|28.45%|}$
	qemu-arm64				30000	54307	${\color{red}\verb|81.02%|}$
	qemu-riscv64				30000	32017	${\color{orange}\verb|6.72%|}$

Machine shutdown time

Base	Platform	Suite	Task	Units	Baseline	Actual	Delta
24.04	qemu-amd64	smoke	test-boot	milliseconds	3500	3044	-13.03%
	qemu-arm64				3500	3187	-8.94%
	qemu-riscv64				3500	3063	-12.49%

Disk size (compressed)

Base	Platform	Suite	Task	Units	Baseline	Actual	Delta
24.04	qemu-amd64	smoke	test-boot	bytes	21821884	21824805	0.01%
	qemu-arm64				25517168	25521977	0.02%
	qemu-riscv64				34285468	34288386	0.01%

Disk size (decompressed)

Base	Platform	Suite	Task	Units	Baseline	Actual	Delta
24.04	qemu-amd64	smoke	test-boot	bytes	31351390	31359612	0.03%
	qemu-arm64				34602031	34605318	0.01%
	qemu-riscv64				67672166	67671298	-0.00%

PID1 binary size

Base	Platform	Suite	Task	Units	Baseline	Actual	Delta
24.04	qemu-amd64	smoke	test-boot	bytes	11464888	11473080	0.07%
	qemu-arm64				10813624	10813624	0.00%
	qemu-riscv64				10879160	10879160	0.00%