blm-instagram-bot icon indicating copy to clipboard operation
blm-instagram-bot copied to clipboard

Enhancement: Scaling with Celery

Open cartfisk opened this issue 5 years ago • 2 comments

I saw that you mentioned looking for assistance with scaling. Using a distributed computing library to assign work to a group of "worker" nodes would be a nice way to implement this. I'm familiar with a Python library called Celery that does just that. The main question is how to set up a release distribution that enables individuals to spin up their own workers easily on whatever computing resources they have available.

Additionally, I imagine querying for and detecting image matches is the most computationally expensive task in this project. With this in mind, a distributed system will require a way to designate "chunks of work" to individual instances/nodes so that repeat queries are avoided where possible.

I'll look deeper into the code to see if this can be easily accomplished with the current setup.

cartfisk avatar Jun 02 '20 20:06 cartfisk

I think ultimately the thing that prevents this from scaling well is instagrams rate limit, which puts comment placing cool-downs on accounts for a semi-considerable time, and there isn't really any ideal get arounds to this apart from having many instagram accounts.

syntax avatar Jun 02 '20 20:06 syntax

Maybe the set up process for a distributed instance could configure the bot to post with the host’s own account or a throwaway?

cartfisk avatar Jun 02 '20 20:06 cartfisk