Automatic remediation updates fail if ComplianceScan is recreated

[montaguethomas]

Maybe this is due to incorrect usage of the operator and CRDs, but we are hitting an issue where automatic remediation updates won't work.

https://github.com/ComplianceAsCode/compliance-operator/blob/7f5d1b9f9d7613dec7ad372b69721ca792ff6ae5/pkg/apis/compliance/v1alpha1/compliancesuite_types.go#L80-L82

https://github.com/ComplianceAsCode/compliance-operator/blob/7f5d1b9f9d7613dec7ad372b69721ca792ff6ae5/pkg/apis/compliance/v1alpha1/compliancesuite_types.go#L29-L33

This is primarily due to the ownerReferences on the ComplianceRemediation objects created.

apiVersion: compliance.openshift.io/v1alpha1
kind: ComplianceRemediation
metadata:
  ownerReferences:
  - apiVersion: compliance.openshift.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ComplianceCheckResult

We've had several times we've needed to delete the ComplianceSuite or ScanSettingBinding objects in order to update settings or resolve an issue with scans failing to complete. By doing this, it triggers the deletion of the owned ComplianceRemediation objects. Because deletion of the ComplianceRemediation does not remove the applied remediation (MachineConfig, KubeletConfig, etc), new scans will not trigger findings and thus won't create new ComplianceRemediation. Additionally, even if the MachineConfig objects were removed, the created files on disk would still exist and are not removed/cleaned up.

Would it be possible to remove the ownerReferences on either the ComplianceRemediation or ComplianceCheckResult objects, thus leaving them behind when a ComplianceScan is deleted. By adding additional labels to these objects with details on which ssg rule and remediation each originate from, it could be possible to then relink the objects to a newly created ComplianceScan object.

[rhmdnd]

I recently hit this issue when writing end-to-end tests that clean up stale compliance check results when a scan is rerun.

I think we need to figure out if we can safely decouple the remediation ownership from compliance check results.