natural-gradients
natural-gradients copied to clipboard
wiseodd
Collection of algorithms for approximating Fisher Information Matrix for Natural Gradient (and second order method in general)
https://github.com/wiseodd/natural-gradients/blob/7d51f19d315cb393a52a371162aaf2e27d20dbaa/numpy/toy/full_fisher.py#L47 I have no idea how do you compute fisher information matrix. Specifically, I don't know how do you compute p(x|θ) without using a prior in your data. Can you...
I looked at the full_fisher numpy file and tried to change some parameters but it doesn't work. For example, when the `X0` and `X1` are changed as follows, the loss...
Hi @wiseodd, thanks for great implementation! I wonder whether this can be generalized to a simple feed-forward network. In your code, [the gradient wrt the parameters](https://github.com/wiseodd/natural-gradients/blob/master/numpy/toy/full_fisher.py#L47) are computed directly. For...
In the pytorch implementation of kfac, G1_ is computed as: G1_ = 1/m * a1.grad.t() @ a1.grad However, the a1.grad is different from the a_1 in (1) of kfac's paper....
