stan-dev
Add LOO Difference Plot
Codecov Report
Merging #178 (a0417dd) into master (0117858) will decrease coverage by
2.31%. The diff coverage is0.00%.
@@ Coverage Diff @@
## master #178 +/- ##
==========================================
- Coverage 95.15% 92.84% -2.32%
==========================================
Files 28 29 +1
Lines 2663 2697 +34
==========================================
- Hits 2534 2504 -30
- Misses 129 193 +64
| Impacted Files | Coverage Δ | |
|---|---|---|
| R/loo_difference_plot.R | 0.00% <0.00%> (ø) |
|
| R/effective_sample_sizes.R | 88.18% <0.00%> (-9.45%) |
:arrow_down: |
| R/importance_sampling.R | 87.23% <0.00%> (-5.32%) |
:arrow_down: |
| R/loo.R | 93.82% <0.00%> (-4.53%) |
:arrow_down: |
| R/loo_moment_matching.R | 98.42% <0.00%> (-0.79%) |
:arrow_down: |
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact),ø = not affected,? = missing dataPowered by Codecov. Last update 0117858...a0417dd. Read the comment docs.
@avehtari I've added an example of a plot using the IQ dataset from rstanarm:
The first model only has a constant intercept; the second model also includes a term for the mother's IQ. At the edges, the constant-based model is more accurate than the full model, while the IQ-based model is stronger in the center. This seems counterintuitive, until you realize that this reveals there is significant measurement error in IQ tests: extreme values are probably caused by mismeasurement and need to be adjusted for regression to the mean.
Other examples:
The plots looks great! Thanks for making this PR!
Thanks! Do you happen to know if the spatial data from the Bayesian visualization paper can be added to the loo package? I think it's a good example, but if it can't, I can remove it.
Sorry for the delay in responding to this. Regarding the issue of including the data from the visualization paper, I think we should avoid adding it to the package but I think there's still a way we can still use it. Basically, we should use a different example in the Examples section in the documentation, but then we can add an example using the visualization paper data in one of the vignettes. This works because in the vignette we're allowed to download data that isn't included in the package. For example, we do this already in one of the vignettes with some other data:
https://github.com/stan-dev/loo/blob/01178587e101c4e06e5ab078acdf69ab272a75f2/vignettes/loo2-with-rstan.Rmd#L95-L96
So we could do something similar and download the visualization paper data from https://github.com/jgabry/bayes-vis-paper. @ParadaCarleton What do you think about that option?
Sorry for the delay in responding to this. Regarding the issue of including the data from the visualization paper, I think we should avoid adding it to the package but I think there's still a way we can still use it. Basically, we should use a different example in the Examples section in the documentation, but then we can add an example using the visualization paper data in one of the vignettes. This works because in the vignette we're allowed to download data that isn't included in the package. For example, we do this already in one of the vignettes with some other data:
https://github.com/stan-dev/loo/blob/01178587e101c4e06e5ab078acdf69ab272a75f2/vignettes/loo2-with-rstan.Rmd#L95-L96
So we could do something similar and download the visualization paper data from https://github.com/jgabry/bayes-vis-paper. @ParadaCarleton What do you think about that option?
Sounds good! I've removed the example. Where do you think I should put this -- should I add it to an existing vignette or create a new one?
Where do you think I should put this -- should I add it to an existing vignette or create a new one?
Hmm, on the one hand I think it could be nice to have it in the main introduction vignette https://mc-stan.org/loo/articles/loo2-example.html. On the other hand, this would be using a totally different data set than the example in that vignette, so that's not ideal and maybe a new vignette is preferable.
I think if we go the route of a new vignette then it shouldn't just be about this one plot but rather about visualizing loo output in general (e.g., this plot, the Pareto k diagnostic plot, perhaps some of the loo related plots in bayesplot, etc.). But that would be more work than just adding this to an existing vignette so it depends how much you feel like working on this (no pressure!).
@jgabry btw, I think we should merge this, unless there's some change you'd like to see. I might be able to get to building a vignette before classes start again, but given that I'm focused on adding new features to ParetoSmooth.jl I'm not sure I'll be able to. We can probably add this to some other vignette later.