Add @atomic for atomic operations

[dmed256]

Use appropriate backend locks to support:

• [ ] Update expressions (+=, -=, *=, /= updates)
• [ ] Block statements (critical regions)

Atomic expressions

Inlined atomic

@atomic arr[i] += value;

arr[i] += value @atomic;

Single update-statement block

@atomic {
    arr[i] += value;
}

Block statements

@atomic {
    arr1[i] += value;
    arr2[i] -= value;
}

[pdhahn]

I hope @atomic blocks (which I interpret to correspond to critical sections) are reliably implemented sooner than later. As far as I am concerned, it would be a really desirable feature to have.

Also from (admittedly) casual browsing, I worry some OCCA issues mentioning atomic blocks may presume

@atomic
{
    arr1[i] += value;
    arr2[i] -= value;
}

is equivalent to

@atomic arr1[i] += value;
@atomic arr2[i] -= value;

Unless I misunderstand how @atomic is translated by modern OCCA, I think these are not equivalent from the perspective of wanting an "atomic block" in OCCA to correspond to an actual "critical section". Or is there something I am missing about issues with atomics on GPU's via OpenCL or CUDA? Obviously, I have not pursued any evaluation of this yet to discover the answer (sorry).

A spinning atomic test-and-set (as in Dr Hans Boehm's old atomics library that I use routinely in my CPU multi-threaded code) is a simple way to implement a critical section. I do not know if anything like that is supported with OpenCL/CUDA on GPU's(?) that could be be leveraged by OCCA as a way to implement atomic blocks.

[kris-rowe]

Support for atomics is partially implemented but needs to be reviewed and made consistent across different backends. I have tried to summarize the state of things below.

Unless I misunderstand how @atomic is translated by modern OCCA, I think these are not equivalent from the perspective of wanting an "atomic block" in OCCA to correspond to an actual "critical section".

I agree. Intuitively, one would expect a block @atomic to be a shortcut for applying atomic to each statement within the block; however, even this interpretation could cause confusion (with little gained). If critical sections are needed, it would make sense to add a separate @critical attribute for this purpose.

The topic of atomics in general merits discussion at the OCCA TAF. I will add it to the agenda for the next meeting.

Status Summary

Mode	Single Statement	Block Statement	Operations
OpenMP	Yes	Yes	=, +=, -=, *=, /=
CUDA	Yes	No	+=, -=
HIP	Yes	No	+= -=`
SYCL (DPCPP)	Yes	Yes	=, +=, -=, *=, /=
OpenCL	No	No
Metal	No	No

[pdhahn]

Yes I believe you are correct: a distinct @critical attribute is the right way to go for a critical section. So keep @atomic single-statement-oriented. Having the "atomic block" construct as a SIMD-like "scatter meta-operation" of @atomic in attribute space is not immediately intuitive to me personally and maybe not to others as well. A misunderstanding of its semantics by those who use it is not unlikely. I recommend removing it (atomic block) because of that argument, though not because I believe it is a bad idea per se. I perceive it as sort of "nifty" but something that doesn't bring that much to the table even if well understood and properly used by the programmer.

Again, what I am really interested in is the @critical and I hope its development can become a priority for the OCCA team. Of course @atomic is especially convenient for accumulation like counts or simple scalar reductions. But @critical would obviously be far more useful in general.

Thanks also for the valuable status summary on @atomic! Good stuff.