risk of getting empty messages
sqs can return empty messages if the message number is low (<1000), maybe this explains that why the last few hundred tasks takes longer time.
this package set the wait time to be 0, and could potentially get empty messages. https://github.com/seung-lab/python-task-queue/blob/master/taskqueue/aws_queue_api.py#L60
here is the documentation of the long polling: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/working-with-messages.html#setting-up-long-polling
as suggested, simply setting the waittime to 20 seconds should fix this.
Wait time has been set to 20 sec since https://github.com/seung-lab/python-task-queue/commit/a355d042b8f126947d458a4e82f69a0d3ebd79a8