Profiles and Controls with invalid rule IDs build successfuly when they shouldn't

[yuumasato]

Description of problem:

The build system silently ignores invalid rule IDs in profiles and controls.

SCAP Security Guide Version:

Latest master (ab510556f587bf0140da78593c46b86450778ab5)

Operating System Version:

Any

Steps to Reproduce:

1. Checkout master
2. Add a invalid IDs to a Profile and build
3. Add a invalid IDs to a Control and build

Actual Results:

Build succeeds without error, warning or traceback.

Expected Results:

• An error, warning or traceback about the invalid ID.
• Last commit where invalid ID in a Profile caused a traceback was 4c248d2a050a3d91f62c8a4dec3c54b332a2a1ba. ( Note: At this commit, invalid IDs in Controls were not being correctly caught, maybe it was never detected) With an invalid ID in a Profile, this was the traceback:

./build_product -j8 rhel8
...
[2/59] [rhel8-content] compiling profiles
FAILED: rhel8/profiles 
cd /home/wsato/git/parallel-content/build/rhel8 && /usr/bin/cmake -E make_directory /home/wsato/git/parallel-content/build/rhel8/profiles && env PYTHONPATH=/home/wsato/git/parallel-content /usr/bin/python3 /home/wsato/git/parallel-content/build-scripts/compile_profiles.py --controls-dir /home/wsato/git/parallel-content/controls --build-config-yaml /home/wsato/git/parallel-content/build/build_config.yml --product-yaml /home/wsato/git/parallel-content/products/rhel8/product.yml -o /home/wsato/git/parallel-content/build/rhel8/profiles/{name}.profile
Not building profile from /home/wsato/git/parallel-content/products/rhel8/./profiles/ospp-mls.profile: documentation not complete and not a debug build
Not building profile from /home/wsato/git/parallel-content/products/rhel8/./profiles/stig.profile: Unable to find rule 'file_permissions_var_log_messages_error'
Traceback (most recent call last):
  File "/home/wsato/git/parallel-content/build-scripts/compile_profiles.py", line 77, in <module>
    main()
  File "/home/wsato/git/parallel-content/build-scripts/compile_profiles.py", line 70, in main
    profiles[pname].resolve(profiles, controls_manager)
  File "/home/wsato/git/parallel-content/ssg/build_yaml.py", line 435, in resolve
    raise RuntimeError(msg)
RuntimeError: Profile stig_gui extends profile stig, but only profiles ['anssi_bp28_enhanced', 'anssi_bp28_high', 'anssi_bp28_intermediary', 'anssi_bp28_minimal', 'cis', 'cis_server_l1', 'cis_workstation_l1', 'cis_workstation_l2', 'cjis', 'cui', 'e8', 'hipaa', 'ism_o', 'ospp', 'pci-dss', 'rht-ccp', 'standard', 'stig_gui'] are available for resolution.
ninja: build stopped: subcommand failed.

Additional Information/Debugging Steps:

• After #7716 the build doesn't complain about invalid IDs.
  • A check for invalid rules was removed without replacement: https://github.com/ComplianceAsCode/content/commit/20491c7c5876dc335b62140b6a43100116f18458#diff-7bdd84322da01c2b17cc451968f9b7eb024114feec6b64e25fd264710a119424L235 (expand diff for ssg/build_yaml.py)
  • Invalid IDs may be ignored in resolve_selections_with_rules(): https://github.com/ComplianceAsCode/content/commit/20491c7c5876dc335b62140b6a43100116f18458#diff-7bdd84322da01c2b17cc451968f9b7eb024114feec6b64e25fd264710a119424R551-R552.
    • This is particularly a problem because resolve_controls() may add invalid invalid rule IDs. The code doesn't have a way to distinguish invalid rule IDs from rules not available to the product. This is partly caused by the BuildLoader discarding all rules not applicable, due to incompatible prodtype.
    • Another effect of resolve_selections_with_rules() is that it drops out invalid IDs and the other check for invalid IDs never triggers: https://github.com/ComplianceAsCode/content/blob/master/ssg/build_yaml.py#L669

[yuumasato]

Ping @matejak @jan-cerny I'm not sure how to fix this issue.

One way I can think of is for the BuildLoader to also keep track of all rules, not just the applicable ones. So that the build system can warn of invalid rule IDs, but not get tricked by valid rule IDs if a control adds a rule that is not applicable to the product being currently built.