lyft
Excessive memory usage
The library uses a excessive amount of memory, at least 2GB reading from cached data and as much as 3.5GB reading fresh data from the API. This results in excessive initialization times even in cached mode (30 seconds or more).
The net effect results in this library being much less useful that it might otherwise be. For example, it's nearly impossible to use this from Lambda given the hard memory cap of 3GB and high cost of such configurations.
Much of this is surely a side effect of the limited nature of the underlying AWS Pricing data and APIs, but I believe it can be addressed. I'm going to look into alternative methods for both pulling down the initial data as well as breaking up the cache into shards of some flavor to avoid the need to load the entire json blob into memory at once.
Agreed that it'd be great to cut down on memory usage! Have you been able to give v2.0.0 a try yet? That should hopefully help a bit, but open to PRs to improve this further.
Hi @garrettheel
Issue
The memory issue still exists and gets worse everytime AWS adds more instance types I guess ;-). Right now my script uses up to 9.5GB of memory. By tracing it with py-spy I can see the CPU is heavily used to decode JSON and I suspect the memory is used by all the dict objects resulting. While it was bearable a year ago now my i7 / 16GB Macbook pro 2015 comes to almost to halt for 30sec because of my script running with the following line and often I have to kill firefox to have enough memory to run it otherwise it fails with a MemoryError
Code
offerings = conn.describe_reserved_instances_offerings(MaxDuration=MaxRiDuration * 2592000,InstanceTenancy='default')
Questions
Is there any possible workaround to use less memory? and as a bonus less CPU... Other than buy a new laptop ;-)
Environment
- Python 2.7.12+ (Debian)
- awspricing (2.0.3)
