ACS(Architecture Compliance Suite) Status

[bitboom]

Status

REGRESSION REPORT: 
==================
   TOTAL TESTS     : 51 (8 skipped)
   TOTAL PASSED    : 30
   TOTAL FAILED    : 18
   TOTAL SKIPPED   : 2
   TOTAL SIM ERROR : 1

Details

no.	category	testcase	Passed	Partially Passed	Failed	Assignee
1	interface	cmd_granule_delegate	O			https://github.com/Samsung/islet/pull/135
2	interface	cmd_granule_undelegate	O			https://github.com/Samsung/islet/pull/148
3	interface	cmd_realm_create	O			https://github.com/Samsung/islet/pull/155
4	interface	cmd_realm_activate	O			https://github.com/Samsung/islet/pull/183
5	interface	cmd_rtt_read_entry	O			https://github.com/Samsung/islet/pull/156, https://github.com/Samsung/islet/pull/193, https://github.com/Samsung/islet/pull/201
6	interface	cmd_rtt_init_ripas	O			https://github.com/Samsung/islet/pull/193, https://github.com/Samsung/islet/pull/205
7	interface	cmd_rtt_create	O			https://github.com/Samsung/islet/pull/193, https://github.com/Samsung/islet/pull/205
8	interface	cmd_data_create	O			https://github.com/Samsung/islet/pull/183, https://github.com/Samsung/islet/pull/193, https://github.com/Samsung/islet/pull/202
9	interface	cmd_data_create_unknown	O			https://github.com/Samsung/islet/pull/193, https://github.com/Samsung/islet/pull/212
10	interface	cmd_rtt_map_unprotected	O			https://github.com/Samsung/islet/pull/220, https://github.com/Samsung/islet/pull/156
11	interface	cmd_rec_create		O (35/36)		https://github.com/Samsung/islet/pull/185
12	interface	cmd_rec_enter	O			https://github.com/Samsung/islet/pull/198
13	interface	cmd_rmi_version	O			https://github.com/Samsung/islet/pull/154
14	interface	cmd_rsi_version	O			https://github.com/Samsung/islet/pull/154
15	interface	cmd_realm_config		O		https://github.com/Samsung/islet/pull/154
16	interface	cmd_ipa_state_get	O			https://github.com/Samsung/islet/pull/154
17	interface	cmd_ipa_state_set			O	@jinbpark
18	interface	cmd_multithread_realm_up			O
19	interface	cmd_multithread_realm_mp			O
20	interface	cmd_secure_test			O
21	memory management	mm_ripas_change	O
22	memory management	mm_ripas_change_reject	O
23	memory management	mm_ripas_change_partial	O
24	memory management	mm_hipas_assigned_ripas_empty_da			O	@zpzigi754
25	memory management	mm_hipas_destroyed_ripas_ram_da			O
26	memory management	mm_hipas_assigned_ripas_empty_ia			O
27	memory management	mm_hipas_assigned_ia			O
28	memory management	mm_hipas_assigned_da	O			https://github.com/Samsung/islet/pull/197
29	memory management	mm_unprotected_ipa_boundary	O			https://github.com/Samsung/islet/pull/197
30	memory management	mm_protected_ipa_boundary			O	@nook1208
31	memory management	mm_gpf_exception			O
32	memory management	mm_rtt_translation_table	O			https://github.com/Samsung/islet/pull/197
33	memory management	mm_rtt_fold_assigned			O
34	memory management	mm_rtt_fold_unassigned			O
35	memory management	mm_rtt_fold_u_ipa_assigned			O
36	memory management	mm_rtt_fold_destroyed			O
37	memory management	mm_hipas_destroyed_ripas_ram_ia			O
38	memory management	mm_hipas_unassigned_ripas_empty_da			O
39	memory management	mm_hipas_unassigned_ripas_empty_ia			O
40	memory management	mm_hipas_destroyed_ripas_empty_da			O
41	memory management	mm_hipas_destroyed_ripas_empty_ia			O
42	memory management	mm_hipas_unassigned_ripas_ram_da			O
43	memory management	mm_hipas_unassigned_ripas_ram_ia			O
44	memory management	mm_feat_s2fwb_check_1	O			https://github.com/Samsung/islet/pull/197
45	memory management	mm_rtt_level_start	O			https://github.com/Samsung/islet/pull/193
46	exception	exception_rec_exit_wfe			O
47	exception	exception_rec_exit_wfi			O
48	exception	exception_rec_exit_hostcall	O			https://github.com/Samsung/islet/pull/156
49	exception	exception_rec_exit_psci			O
50	exception	exception_realm_unsupported_smc	O			https://github.com/Samsung/islet/pull/156
51	exception	exception_rec_exit_hvc	O			https://github.com/Samsung/islet/pull/156, https://github.com/Samsung/islet/pull/203
52	exception	exception_rec_exit_ripas	O			https://github.com/Samsung/islet/pull/197
53	exception	exception_rec_exit_ia	O			https://github.com/Samsung/islet/pull/197
54	exception	exception_emulatable_da			O
55	exception	exception_non_emulatable_da			O
56	exception	exception_non_emulatable_da_1			O
57	exception	exception_non_emulatable_da_2			O
58	gic	gic_hcr_invalid			O
59	gic	gic_ctrl_list_invalid	O			https://github.com/Samsung/islet/pull/198

Related Issue

• https://github.com/Samsung/islet/issues/146
• https://github.com/Samsung/islet/issues/162
• https://github.com/Samsung/islet/issues/186

[jinbpark]

Note that I confirmed test cases with this PR to eliminate the cache problem. Plus, I'm doing a single test case at a time, instead of a whole.

[zpzigi754]

It is interesting that cmd_realm_config has passed in ACS, because it was malfunctioning (almost like no-op).

[jinbpark]

@zpzigi754 I confirmed what you said. You're right. It seems that ACS test doesn't test the point where we do wrong. I turned cmd_realm_config from Passed to Partially Passed. REALM_CONFIG should be re-implemented according to the spec, regardless of the result of ACS test.

[jinbpark]

@bitboom , @zpzigi754

An important note:

I found that all ACS tests that involve ACS realm payload (e.g., cmd_realm_create doesn't involve a realm payload, but cmd_rec_enter or cmd_rsi_version involve it) use "SHARED REGION" between "Realm" and "Host".

But, currently, "SHARED REGION" doesn't work as @zpzigi754 said before. So, we need to wait for #156 that solves this sharing issue to get merged, in order to handle such ACS tests. Before that, we should look at other ACS test cases first.