feat: locking jobs on a per tenant basis

[ashleyww93]

Is your feature request related to a problem? Please describe.

Some APIs (Shopify) have call limits in place, in Shopify's case they use leaky bucket.

When you have complex queries you can rapidly a lot of the credits, and you must wait for them to refill, this is fine, however, if you have more than one job for a tenant trying to call Shopify's endpoints, both have a chance to end up stuck because they are both exhausting the credits.

These could be completely different jobs, and so they don't belong in the same queue.

Describe the solution you'd like to see

I would like trigger.dev to be able to support running one job per tenant.

When sending an event, you could optionally send a tenantId, which would ensure that this event would only be triggered if the tenantId is not already in use. If the Id was already in use, it should simply join a queue and trigger the oldest events first.

If an event does not provide a tenantId, then it should continue to work as normal.

Describe alternate solutions

I looked into the store, but it looks like the job would be started over and over, where ideally the triggers should never arrive until the job can start.

In my current solution(a custom implementation) we use redis locks.

Additional information

I posted a discussion about this 3 weeks ago: https://github.com/triggerdotdev/trigger.dev/discussions/787