stan-dev
bug: inconsistent behaviour of `mcse_quantile`
the mcse quantile function does break down for quantile 100%, but works for the minimum:
I would expect it to work just fine for the maximum, no... but right now I get NA instead.
So the problem is that .ess is NA for constant inputs. We compare x <= quantile(x, prob) which is TRUE for at least one value for the 0% quantile (and FALSE otherwise; i.e. not constant), while the same expression is TRUE for all values for the 100% quantile, hence creating the NA. @avehtari any ideas how to deal with that case?
- I think the point is that empirical quantile(,0) is not minimum and empirical quantile(,1) is not maximum.
- Overall extreme quantiles with only one or a few draws on the other side of the threshold have bad MCSE estimates. This is due to a) not having enough observations for empirical estimates, b) made even worse if the draws are dependent as is likely with MCMC.
- Should we add a warning for the case in point 2? (for a moment we also had a version that produced NA if any of the chains was constant, but caused too many NAs)
- If there is a need for MCSEs for extreme quantiles, we can use extreme value analysis to provide better estimates. However, this is not easy to make the default, as this would need to make some assumptions about the tail, but if the user knows what they are doing it would be easy to implement useful functions.
@wds15 in light of our above comments, do you have a preference if/how we shall change the current behavior?
The extreme quantiles of a posterior are not a minimum nor a maximum to me. What I would like to see as a user is a continuous output if possible. So, if I approach one of the extremes, then the mcse should be continuously smooth to some degree.
The extreme quantiles of a posterior are not a minimum nor a maximum to me.
Are you talking about real quantiles or sample quantiles (returned by quantile())?
What I would like to see as a user is a continuous output if possible. So, if I approach one of the extremes, then the mcse should be continuously smooth to some degree.
I would like to see that, too, but as I commented above, the MCSE estimates based only on the draws are inherently unstable as there are only one or few draws to use to make that estimate. Except for the NA, I think we can't change the default behavior, as quantile() function itself is not including additional tail model information. It would be possible to improve the behavior by providing quantile estimates based on modeling the tail, e.g., with generalised Pareto distribution, which the user could choose with an additional option (understanding that use of tail model can fail if the tail is not well modeled by that model)
I just fixed the NA issue when prob = 1 such that it always returns the same value as prob = 1 - e with e being arbitrarily small. @wds15 does that resolve your initial issue?
@avehtari quantile returns sample quantiles to me. Maybe we can have an mcse for min or max? That could be interesting, but would probably need to be defined in terms of some tail probability being beyond the so defined minimum/maximum? Anyway, that's off-topic.
@Paul-buerkner What you describe sounds like the right thing to me. So, yes that's great. Thanks!
Maybe we can have an mcse for min or max? That could be interesting, but would probably need to be defined in terms of some tail probability being beyond the so defined minimum/maximum? Anyway, that's off-topic.
Off-topic, but we can have distributions for min and max if make an assumption about the tail shape (usual extreme value analysis thing). If you wish, you can make a new issue
Okay, I will close to issue now, but feel free to open a new one for the new discussion points that arose above.