Add capability to discard duplicate jobs with concurrency configuration

[joelzwarrington]

Closes https://github.com/rails/solid_queue/issues/176

Add ability to discard duplicate jobs instead of becoming blocked. When configuring concurrency_limits with at_limit: discard jobs scheduled above the concurrency limit are discarded and not executed.

Implementation Details

• adds class_attribute concurrency_at_limit and at_limit option to concurrency_limits
• adds methods to Semaphore to detect when a job would be above the concurrency limit
• adds logic to discard a job on dispatch when at concurrency limit

[joelzwarrington]

Thanks for the review @rosa ❤️, I'll review your comments more thoroughly and make some changes. 😊

[rajeevriitm]

I am really in need of this feature. As its not yet available , could you help me in implementing this ? I have a ContinuousSearchJob thats run with recurring.yml . Sometimes the job takes longer and I dont want another job to be added to the queue unless the existing one is finished. Could you help me with the best way I can implement this currently?

[joelzwarrington]

@rajeevriitm if you have a look at the read me you should find concurrency controls are available for you to use.

The only difference being that jobs will always be blocked, so you may get several jobs that are awaiting execution.

[joelzwarrington]

@rosa any updates on your review here?

[rynonl]

@rosa Just pinging this. Would love to see this functionality get merged!

[danlo]

@rosa +1

[rosa]

Hey, sorry everyone! I've been crazy busy at work in a bunch of unrelated things, but I'm planning for at least a day of Solid Queue work to catch up with all issues and PRs very soon. Thanks y'all for your patience and sorry for the delay!

[joelzwarrington]

Thanks for the review. Gonna have a look and implement any feedback this week

[joelzwarrington]

It'd be a good idea to update https://github.com/rails/rails/blob/c049370882474acffbc8fd5be458e6ecd1d7d1cb/guides/source/active_job_basics.md?plain=1#L346 when merging this as I've changed SolidQueue::Job::EnqueueError to inherit from ActiveJob::EnqueueError.

[joelzwarrington]

@rosa, pull request is ready for you review again! Let me know what you think.

If you're on-board with these changes, I'll also add a PR to update the RoR edge guide.

[marelons1337]

Would love to see that implemented, defo +1

[joelzwarrington]

👋 @rosa any idea when I should expect a re-review? I don't mind if you go a different direction but would love to get this type of functionality implemented

[rosa]

@joelzwarrington sorry for the delay 😓 I've been doing this in my free time and didn't have time over the weekend. I hope to get to it sometime this week or next week, when I'll have a few hours dedicated to Solid Queue 🤞 It's top of my list for SQ work. I'd like to incorporate the idea from #586 because I think that simplifies a lot, but I need to do that carefully to ensure bulk enqueuing works as it should.

[rosa]

Cherry-picked a bunch of commits from here and created https://github.com/rails/solid_queue/pull/594, so I'm closing this one. Thanks a lot! 🙏