facebookarchive
use generators instead of pre-computing all the samples into the memory
helps to save huge amount of RAM while preserving performance
It appeared impossible for me to use the lib on a machine with 64Gigs of RAM because it ran out of memory. This simple fix made memory footprint neglectable.
Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please sign up at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need the corporate CLA signed.
If you have received this in error or have any questions, please contact us at [email protected]. Thanks!
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!
Hi @jbinary ! Thanks for the commit! It looks like izip is not supported by python3 (perhaps being automatically handled by zip - https://stackoverflow.com/questions/32659552/importing-izip-from-itertools-module-gives-nameerror-in-python-3-x). Can you run unit tests for python3 and confirm your solution works? I am getting test failure errors around izip for python3 runtime. Thank you!
hey guys, I have no any chance to see what went wrong here, could you give me a clue maybe?