igmhub
Update utils.py with bootstrap covariance estimate
I wrote a simple bootstrap estimator for the covariance matrix for test purposes.
@p-slash Do you need a review for this PR ? The function is good but it is not usable without a bin script, or linking the new functions into the code, do you plan to do that ?
The bootstrap covariance seems to work fine. The results are close to the original covariance estimate. It is somewhat slower than I expected, but time should not be a problem. I just created an option to use bootstrap covariance in picca_export.py
The default is 10000 bootstrap in picca_export.py, which can be changed if desired.
1D plot compares the diagonals of the current covariance to the bootstrap covariance. The differences can be attributed to noise in estimation. 2D plot is the relative difference between two matrices (delta C / v v^T where v is the mean diagonal of both estimates). Here, we see a structure in scale dependence.
Hi @p-slash, this is putting your estimator of the covariance as the default now, shouldn't we do the opposite ?
I was trying to slip this in as the default 👿 . Seriously though, let's make it the default 😁 I suppose someone should test it first...
@p-slash, I would be happy to make it the default, but we should test it first. I would start with the stack of mocks (both IFAE-QL and Saclay) and check that the fits are consistent (although they will probably not be identical). @Laura-Casas, @HiramHerrera would either of you be interested in helping with this?
The best would be to redo the uncertainty validation we did for DR2 BAO. This would involve computing the bootstrap covariance for all mocks, refitting them, and computing the statistics of the pull distributions. However, given that this is quite a bit more work, I would say we should just do this for DR2 full-shape directly. So if we're happy with the tests on the stacks of mocks, we can merge it as the default new method.
Would it be ok if I implemented the bias correction to the sample variance estimator? I will revert back the default behavior of doing bootstraps. Then, let's merge this branch if that's ok.