GLJeff

In sm_simple.py, SM-G-SUM and SM-G-ABS scaling differ by sz^2

1

Note that torch.autograd.backward() calculates the sum of gradients in all states (at least in 0.4.1 https://pytorch.org/docs/stable/autograd.html?highlight=backward#torch.autograd.backward) SM-G-SUM feeds backward() outputs of 1 and then uses the returned gradients unaltered (ie...