stan ADVI diagnostic

Summary:

Add PSIS diagnostic for ADVI .

Description:

This paper proposes PSIS diagnostic for ADVI: Yuling Yao, Aki Vehtari, Daniel Simpson, and Andrew Gelman (2018). Yes, but Did It Work?: Evaluating Variational Inference https://arxiv.org/abs/1802.02538

Currently ADVI samples N draws from the approximate distribution and outputs those. In addition lp__ column is inserted, but it has all 0's (it seems stansummary requires lp__ column).

To follow the current design with stansummary, what we propose is

[x] after ADVI finishes, compute lp__ and lg__ (log density of of the approximation) in stan/src/stan/variational/advi.hpp
[x] output lp__ and lg__ along with parameter sample (for lp__ provide values instead of 0's and add lg__ column). Something needs to be added also in stan/src/stan/services/experimental/advi/meanfield.hpp
[ ] add algorithms for computing khat and neff in services
[ ] Given the output we can compute diagnostic afterwards, e.g., using stansummary for CmdStan
for simplicity we would implement this first for CmdStan (needs separate issue), ie, modify stansummary to output diagnostics cmdstan/src/cmdstan/stansummary.cpp

Note that initially we start just computing the diagnostics, but eventually we can use diagnostics to improve the algorithm.

Feb 22 '18 17:02 avehtari

@yao-yl is working on this, but I can't assign this issue for him

Feb 22 '18 17:02 avehtari

I do not know why I cannot get assigned or self-assigned. But yes, I am working on this. I have implemented this in cmdstan and will try to add it in rstan.

Feb 22 '18 20:02 yao-yl

I think you need to be added as a collaborator to be assigned---I think @syclik and @seantalts have the permissions to do that for you.

Feb 23 '18 17:02 bob-carpenter

Weird! I'll look into it.

Feb 23 '18 17:02 seantalts

@yao-yl I was thinking about assigning myself, but it appears you've already implemented this:

I have implemented this in cmdstan

We can do separate PR's for each interface.

Want to drop me the code and I can go ahead and integrate it and write appropriate tests?

Jul 22 '19 09:07 drezap

I just updated the issue. 2 first things are in Stan. Next thing would be to implement khat and n_eff in services.

Jul 22 '19 12:07 avehtari

To compute khat I'm using Zhang and Stephens section 4?

Jul 29 '19 18:07 drezap

The reference I know for ADVI eval with k-hat is Yao et al..

Jul 29 '19 21:07 bob-carpenter

You can use the R code as a reference implementation https://github.com/stan-dev/loo/blob/master/R/gpdfit.R It follows Zhang and Stephens , but uses numerically more stable computations plus weakly informative prior.

Jul 30 '19 19:07 avehtari

and then for Neff?

Jul 30 '19 21:07 drezap

For Neff see the variance formula which is last equation on page 4 in https://arxiv.org/abs/1507.02646 and for the reference code see https://github.com/stan-dev/rstan/blob/develop/rstan/rstan/R/stanmodel-class.R#L288 lines 288-304

Jul 31 '19 19:07 avehtari