[Request] Queue throttling

[Salakar]

Firstly, disque is great! \o/

Secondly are there any plans to implement queue throttling or some variation of it. I'll explain:

For example: X public web API has a limit of 100 http requests per 10 secs, and I use a single job to consume one of these requests.

Now normally in Redis I would achieve this using a mix of EXPIRE and INCR - so I set the ttl to 10 seconds and INCR on each successful job run.

Here's a rough redis lua implementation of the above, to demonstrate:

  --[[
  key 1 -> key name - ip, user id or some unique key to throttle X by
  arg 1 -> limit
  arg 2 -> seconds

  returns {
    throttled: ->  1 if should throttle, 0 if still within limit
    remaining: ->  how many reqs left until throttled,
    ttl:       ->  seconds remaining until limit resets
  }
  ]]

  local count = redis.call('INCR', KEYS[1])
  local ttl = redis.call('ttl',KEYS[1])

  if count == 1 or ttl == -1 then
    redis.call('EXPIRE', KEYS[1], ARGV[2])
  end

  if ttl == -1 then
    ttl = tonumber(ARGV[2])
  end

  if count > tonumber(ARGV[1]) then
    return {1,0,ttl}
  end

  return {0,tonumber(ARGV[1]) - count,ttl}

On a single queue job consumer this wouldn't be a problem really, could just keep counts locally, but I currently run a farm of queue workers (using redis), so would be no easy way to maintain the counts across all the servers without using redis alongside disque.

On ADDJOB I see there is DELAY and MAXLEN which is a good start though :+1:

Possible solutions:

1. Allow storing arbitrary data onto queues, then let the clients handle limits / any other use cases for storing queue specific data.
2. Allow storing arbitrary data on disque, same as above but not queue specific - probably better due to queues getting deleted when empty. Doesn't need to be another redis, just the basic get / set sort of stuff?
3. Native feature of disque?

Or any other suggestions? Not sure if I'm the only one with this use case though? :hear_no_evil:

Thanks, Mike

[sangupta]

+1

I think rate-limiting for queues should be built-in disque directly - so regardless of the number of messages being put by producers, the system itself throttles the max number of messages to consumers in a given time period.