Queue tagging

[jabinb]

Hey guys

I'm looking to move from SoftLayers message queue to Disque in the future once a stable release is out. However one of the features that I've found incredibly useful from SL's messaging queue is the ability to tag queues.

For example we have several workflows for when a new user registers (e.g. welcome email, schedule on-boarding drip feed etc). Instead of having to persist somewhere the name of each queue and its role we can simply collect up all the queues with the tag user_register and submit the same message to each.

$queues = $messaging->queues(array('user_register'));
foreach ($queues as $q) 
{
    $q->message()
        ->addField('userId', $user->getId())
        ->create();
}

This means if we add a new worker that wants to ingest messages relating to that tag (e.g. a feedback email) we just need to add that tag to the queue and then they will receive those messages.

As I understand this would require making queues somewhat persistent as currently they only exist while they contain messages.

[antirez]

Hello, I wonder what's the difference between the queue tagging approach, and the alternative approach to have just a single queue, named as the tag itself, where the job representation itself states if it's a welcome email, or any other task. The workers just know they need to submit to user_register and all the related tasks will be delivered to them. Thanks.

[mathieulongtin]

A few alternatives:

• For each tag, put a list of queues in a configuration system, for example, a file or a Redis instance
• Put the initial message in "user_register", a daemon listens to "user_register" and copies the message to all the relevant queues.

On Tue, Apr 5, 2016 at 7:12 AM Salvatore Sanfilippo < [email protected]> wrote:

Hello, I wonder what's the difference between the queue tagging approach, and the alternative approach to have just a single queue, named as the tag itself, where the job representation itself states if it's a welcome email, or any other task. The workers just know they need to submit to user_register and all the related tasks will be delivered to them. Thanks.

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/antirez/disque/issues/130#issuecomment-205758678

Mathieu Longtin 1-514-803-8977

[jabinb]

@antirez The problem tagged queues solves is multiple consumers needing the same message but producing different output. A better example would be for an image resizing queue.

• User uploads large image
• Image is required to be served later with different sizes: 40x40, 120x120, 320x320 etc
• Resize job is created with image as the body
• Job is submitted to queues tagged with image_resize
• Workers consuming those queues process their jobs, each asynchronously resizing the image to different sizes.

Currently I'm achieving this using the first method @mathieulongtin mentions by maintaining my own YAML configuration file of the queues with metadata such as tags/delay.