cdowd
Plotting defaults?
It would be nice if there was a good plot default we could create. Two obvious options arise -- plotting what the test stat measures, and plotting the test stat values.
Okay, plotting test stat values requires a full rewrite of the current structure of the permutation_test_builder, because currently it never even has the full vector of test stat distribution in memory.
Plotting what the test stats each specifically measure is a pain for some of them (kuiper, ks) [have to find the relevant points], not sensible for some (AD, dts) [they aren't linear in ecdf], but is easyish for others (cvm, wass). On the other hand, why bother building this functionality? It would provide mostly educational value -- and help in the manual I suppose (here is what KS does, etc).
What we can do (and I've started on already) is code for plotting the two ECDFS and the sampling distribution of ECDFs, which is nice.
Could also plot the difference between the two ECDFs, and a sampling distribution for that? Then it would be straightforward-ish to highlight the different test statistics? (max, max+min, sum absolute values, total area, weighted sum absolutes, weighted area) Not totally convinced this code is worth adding to the package. It would be a lot more code, a ggplot dependency (for no base reason), and a million ways the thing could break. Seems easier to do this, build the pngs, then add those to the manual/readme.
pngs explaining the test stats are in the docs/readme/website.
plot defaults for distribution still seems a good idea. plausibly easily done by making an S3 class, editing the permutation test builder, and making a simple plotting function.
Spitting out the distribution of test stats is probably worthwhile for other reasons