msccl: need to use agent scope acquire/release atomics when there

[wenkaidu]

are workgroup dependencies

Details

Do not mention proprietary info or link to internal work items in this PR.

Work item: "Internal", or link to GitHub issue (if applicable). Internal

What were the changes?
use agent scope acquire/release atomics when there are workgroup dependencies

Why were the changes made?
data corruption may occur due to data coherency issue

How was the outcome achieved?
use agent scope acquire/release atomics when there are workgroup dependencies

Additional Documentation:
What else should the reviewer know?

Approval Checklist

Do not approve until these items are satisfied.

• [ ] Verify the CHANGELOG has been updated, if
  • there are any NCCL API version changes,
  • any changes impact library users, and/or
  • any changes impact any other ROCm library.