High memory consumption

[mgappa]

Describe the bug

When we extract a large number of hosts or vulnerabilities, we see a large memory consumption.

Expected behavior

The memory consumption must be constant and predictable during the data recovery.

Current behavior

Since we have a huge and increasing number of instances, the memory consumption increases to the point of reaching more than 10Go of RAM memory. Type of error we have from AWS step function :

{
  "errorType": "Runtime.ExitError",
  "errorMessage": "RequestId: XXXX Error: Runtime exited with error: signal: killed"
}

Possible solution

During our investigation, we have concluded the function execute is the root cause of the consumption. One way to solve this is to create a generator instead of a list that increases in size at each loop.

Steps to reproduce

1. Have more than 6000 hosts onboarded on primsa with vulnerabilities.
2. Run a script on an environment with a memory size limited to 4 Go.
3. Constat the crash at one point, with a MemoryError or an ExitError

Context

In our enterprise context, we want to create a useful dashboard to animate our team around vulnerabilities and patch these vulnerabilities.

Your Environment

• Version used: The latest eu SaaS version
• Environment name and version: Container image on python:3.11.6-slim-bullseye with prismacloud-api==5.2.8 and aws-lambda-powertools==2.26.0

[mgappa]

I would like to propose a PR for this bug

[SimOnPanw]

@mgappa happy to review your PR when submitted.