feat: add support for shared queue for data processing

[deependujha]

Before submitting

• [ ] Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• [ ] Did you read the contributor guideline, Pull Request section?
• [ ] Did you make sure to update the docs?
• [ ] Did you write any new necessary tests?

What does this PR do?

Fixes #556 Fixes #103

PR review

Anyone in the community is free to review the PR once the tests have passed. If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 67.05882% with 28 lines in your changes missing coverage. Please review.

Project coverage is 79%. Comparing base (de1ccce) to head (6244f1d). Report is 4 commits behind head on main.

Additional details and impacted files

@@         Coverage Diff         @@
##           main   #592   +/-   ##
===================================
  Coverage    79%    79%           
===================================
  Files        41     42    +1     
  Lines      6135   6192   +57     
===================================
+ Hits       4835   4880   +45     
- Misses     1300   1312   +12     

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[deependujha]

closing this pr in favour of #602