[feature request] support multiple queues

[sonots]

I wish that digdag supports multiple queues.

In my company, we setup an AWS autoscaling cluster to run digdag agents as https://techblog.zozo.com/entry/digdag_ha (Japanese) describes.

We want to separate and create multiple clusters to run workflows, for example:

• Tasks of workflow A go to EC2 AutoScaling Cluster A.
• Tasks of workflow B go to EC2 AutoScaling Cluster B.

with infrastructure architecture as:

• digdag master (web ui) <- this is common web UI for cluster A and B.
• digdag agents for cluster A (autoscaling group A)
• digdag agents for cluster B (autoscaling group B)

Currently, to achieve similar goal, we have to separate digdag projects, and create another digdag cluster (and web UI)

[sonots]

Maybe, the design will be:

workflowA.dig

_export:
  queue: A

workflowB.dig

_export:
  queue: B

digdag server (agent) dequeues only from the specified queue:

digdag server --queue A
digdag server --queue B

[sonots]

In addition, supporting priority queues is nice to have.

[hiroyuki-sato]

Maybe, This issue is similar to #727 and #521.

[muga]

SQS usage is one of other candidates.