Fetch AWS EC2 AMIs based on what's in the graph

[ryan-lane]

Description:

Fetching AMIs is difficult, because the default filter options for the API are roughly:

• AMIs I own
• AMIs managed by AWS
• AMIs managed by the community

Fetching all images takes nearly a minute per region, per account, and loads tens of thousands of images, which aren't being actively used.

It would be ideal to fetch the AMIs in a multi-step manner, which introspects the graph to determine which AMIs should be fetched:

1. Load EC2 instances, launch configurations, instance templates, etc.
2. Query the graph to find AMIs in use
3. Fetch/load AMIs based on what's in use.

I'm not sure at this point if cartography is doing graph introspection to determine what to fetch at this point, so I wanted to float the idea first.

[achantavy]

I'm not sure at this point if cartography is doing graph introspection to determine what to fetch at this point, so I wanted to float the idea first.

I believe we do this with IAM policies: there are lots of policies that AWS APIs return to us that don't apply to our environment, so we only sync policies that are attached to a principal: https://github.com/lyft/cartography/blob/41425d1727f10af497472346e94d98d0e1ed66e2/cartography/intel/aws/iam.py#L717-L718

So I agree with your proposal.

[ryan-lane]

Awesome. I may take a stab at this soon, then.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[stale[bot]]

This issue has been automatically closed for inactivity. If you still wish to make these changes, please open a new change or reopen this one.

[ryan-lane]

Not stale. We're working on this.