Cleaner removing too many artifacts

[wblachowski]

Describe the bug I've observed that Cleaner job can remove too many artifacts from db in two cases:

1. Timeout occurrence in AGGREGATE_SUITES_STEP. In spite of timeout, the processing continues and further steps receive incomplete list of referenced artifacts. PR #490 fixes this issue by forcing processing termination after timeout.
2. New suite version (with new artifacts) added to the database while cleaner job is running and the processing passed fetchProjectSuites task. The new artifacts aren't referenced by any suite collected by fetchProjectSuites task and as a result will be removed.

• Version used: AET 3.2.0

To Reproduce

Steps to reproduce the behavior observed in cases mentioned:

1. Timeout occurrence in AGGREGATE_SUITES_STEP:

• Make sure you have some artifacts stored in AET db. It is important that there is at least one artifact you want to keep that is referenced only by some of the suites (not all of them),
• Slow down cleaner execution in GetMetadataArtifactsProcessor by more than one minute. You can do it by adding sleep() function or a breakpoint in process() method of GetMetadataArtifactsProcessor,
• Run the cleaner (by modifying cron expression in config) with dry-run option unchecked.

---

2. New suite version (with new artifacts) added to database after fetchProjectSuites task:

• Slow down execution in GetMetadataArtifactsProcessor by less than one minute (we don't want timeout to happen),
• Run the cleaner with dry-run option unchecked. While the execution is halted in GetMetadataArtifactsProcessor run a new suite which will create new artifacts. Using any mongoDB tool you can observe that the new artifacts appear in the database and then get removed when the execution of cleaner job continues.

Expected behavior Cleaner should not remove artifacts referenced by existing suites.