owl icon indicating copy to clipboard operation
owl copied to clipboard

softmax is not twice-differentiable

Open tachukao opened this issue 6 years ago • 0 comments

The softmax operation in Algodiff is not twice-differentiable.

and softmax ?(axis = -1) x =
      let c = Arr A.(max ~axis (unpack_arr x)) in
      let y = exp (x - c) in
      let a = sum ~axis y in
      y / a

The current implementation involves a call to unpack_arr, which cannot be differentiated.

tachukao avatar Feb 24 '20 14:02 tachukao