datasetinsights icon indicating copy to clipboard operation
datasetinsights copied to clipboard

Reduce memory footprint for captures.py using Dask Bag

Open adason opened this issue 5 years ago • 0 comments

Why you need this feature:

The current implementation of captures.py loads all annotations for all captures frames into memory. This can potentially create memory hog as annotations can greatly increase memory usage.

Describe the solution you'd like:

Explore using Dask Bag to filter annotations that does not need to be loaded into memory. They should only be loaded when Dataset.__getitem__() is called to generate training examples.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

adason avatar Sep 12 '20 01:09 adason