Naming for Reduction Operation

[asroy]

Originally posted by @j4yan in https://github.com/ROCmSoftwarePlatform/composable_kernel/pull/128#discussion_r832741425

IdentityValue

This value is a mathematical property of reduction type, and should be deterministic and not specified by user of reduction

https://docs.oracle.com/javase/tutorial/collections/streams/reduction.html

InitialValue

This value could be specified at runtime, it could be used in multi-stage reduction to hold the accumulated value of previous reduction

Suggested implementation

struct BaseReductionType {};

template<typename T>
struct ReduceSum : public BaseReductionType
{
    static constexpr T kIdentityValue = T{0}; // not modifiable
    
   constexpr T Reduce(T& acc, const T& v)
   {
       acc += v;
   }
};

template<typename ReduceType,
                  typename T>
struct ReductionOperation
{
    T initialValue_ = ReduceType::kIdentityValue; // modifiable
    
   constexpr T SetInitialValue(T& v)
   {
       initialValue_ = v;
   }

   constexpr T InitializeAcc(T& acc)
   {
       acc = initialValue_;
   }

   constexpr T Reduce(T& acc, const T& v)
   {
       ReduceType::Reduce(acc, v);
   }
};

Pointwise operation for reduction should be unified with normal pointwise operation

https://github.com/ROCmSoftwarePlatform/composable_kernel/blob/91d8b7d67ae9dbf8a6e691ea3e17c0b9705c6ba7/include/ck/tensor_operation/gpu/element/element_wise_operation.hpp#L146-L330

A strange use case of UnaryIdentic

[qianfengz]

The two Reduction Operations is not added by my generic reduction codes. It is used by Chao or JianFeng's fusion kernels. Generic Reduction defined Reduction Operator is in file reduction_operator

Later, I will provide P.R to help improving the fusion codes which uses Reduction, the two Reduction Operations will be removed step by step.

[qianfengz]

ReduceSum and ReduceSquareSum were removed in P.R 213

[qianfengz]

We need discussion on how to unify the Elementwise operator used by Reduction and other part of C.K.

[qianfengz]

P.R 274 is related.