Flaky test `test/sample/test_sample.py::test_samples_cis`

[dweindl]

See https://github.com/ICB-DCM/pyPESTO/runs/7522438577?check_suite_focus=true

_______________________________ test_samples_cis _______________________________

    def test_samples_cis():
        """
        Test whether :py:func:`pypesto.sample.calculate_ci_mcmc_sample` produces
        percentile-based credibility intervals correctly.
        """
        # load problem
        problem = gaussian_problem()
    
        # set a sampler
        sampler = sample.MetropolisSampler(
            options={
                'show_progress': False,
            },
        )
    
        # optimization
        result = optimize.minimize(
            problem=problem,
            n_starts=3,
            filename=None,
            progress_bar=False,
        )
    
        # sample
        result = sample.sample(
            problem=problem,
            sampler=sampler,
            n_samples=2000,
            result=result,
            filename=None,
        )
    
        # run geweke test
        sample.geweke_test(result)
    
        # get converged chain
        converged_chain = np.asarray(
            result.sample_result.trace_x[0, result.sample_result.burn_in :, :]
        )
    
        # set confidence levels
        alpha_values = [0.99, 0.95, 0.68]
    
        # loop over confidence levels
        for alpha in alpha_values:
            # calculate parameter samples confidence intervals
>           lb, ub = sample.calculate_ci_mcmc_sample(result, ci_level=alpha)

test/sample/test_sample.py:718: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pypesto/sample/util.py:46: in calculate_ci_mcmc_sample
    lb, ub = calculate_ci(chain, ci_level=ci_level, axis=0)
pypesto/sample/util.py:96: in calculate_ci
    lb, ub = np.percentile(values, percentiles, **kwargs)
<__array_function__ internals>:180: in percentile
    ???
.tox/base/lib/python3.8/site-packages/numpy/lib/function_base.py:4134: in percentile
    return _quantile_unchecked(
.tox/base/lib/python3.8/site-packages/numpy/lib/function_base.py:4383: in _quantile_unchecked
    r, k = _ureduce(a,
.tox/base/lib/python3.8/site-packages/numpy/lib/function_base.py:3702: in _ureduce
    r = func(a, **kwargs)
.tox/base/lib/python3.8/site-packages/numpy/lib/function_base.py:4552: in _quantile_ureduce_func
    result = _quantile(arr,
.tox/base/lib/python3.8/site-packages/numpy/lib/function_base.py:4658: in _quantile
    take(arr, indices=-1, axis=DATA_AXIS)
<__array_function__ internals>:180: in take
    ???
.tox/base/lib/python3.8/site-packages/numpy/core/fromnumeric.py:190: in take
    return _wrapfunc(a, 'take', indices, axis=axis, out=out, mode=mode)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = array([], shape=(0, 1), dtype=float64), method = 'take', args = (-1,)
kwds = {'axis': 0, 'mode': 'raise', 'out': None}
bound = <built-in method take of numpy.ndarray object at 0x7fdfbc362c90>

    def _wrapfunc(obj, method, *args, **kwds):
        bound = getattr(obj, method, None)
        if bound is None:
            return _wrapit(obj, method, *args, **kwds)
    
        try:
>           return bound(*args, **kwds)
E           IndexError: cannot do a non-empty take from an empty axes.

.tox/base/lib/python3.8/site-packages/numpy/core/fromnumeric.py:57: IndexError

[yannikschaelte]

confirmed, ~4% fail rate

[yannikschaelte]

@elbaraim it appears that this is a problem of auto-calculated burn-in, in case the entire array is found to be burn-in. I guess in this case an appropriate error should be thrown, and the test adjusted.

[yannikschaelte]

Also at https://github.com/ICB-DCM/pyPESTO/runs/7650416076?check_suite_focus=true

[elbaraim]

..., and the test adjusted.

@yannikschaelte For the second part, what would you have in mind? Just for the sake of testing the visualization function to, e.g. fix the burn-in index to a given value?

[yannikschaelte]

ok it can ofc happen that geweke says end of chain. if this test is just about running some tests that do not rely on the geweke being some actually reasonable number, then I agree you can just fix it here to some intermediate index.

[dweindl]

Still happening... https://github.com/ICB-DCM/pyPESTO/actions/runs/3280972663/jobs/5402409945