Flaky test_result_from_hdf5_history?

[dweindl]

Fails here https://github.com/ICB-DCM/pyPESTO/runs/3893112578?check_suite_focus=true

I don't think that's related to that commit.

________________________ test_result_from_hdf5_history _________________________

    def test_result_from_hdf5_history():
        problem = create_petab_problem()
        with tempfile.TemporaryDirectory(dir=".") as tmpdirname:
            _, fn = tempfile.mkstemp(".hdf5", dir=f"{tmpdirname}")
    
            history_options_hdf5 = pypesto.HistoryOptions(trace_record=True,
                                                          storage_file=fn)
            # optimize with history saved to hdf5
            result = pypesto.optimize.minimize(
                problem=problem, n_starts=1,
                history_options=history_options_hdf5)
    
            result_from_hdf5 = optimization_result_from_history(fn)
    
            # Currently 'exitflag', 'time' and 'message' are not loaded.
            arguments = [ID, X, FVAL, GRAD, HESS, RES, SRES,
                         N_FVAL, N_GRAD, N_HESS, N_RES, N_SRES, X0, FVAL0]
            for key in arguments:
                if result.optimize_result.list[0][key] is None:
                    assert result_from_hdf5.optimize_result.list[0][key] is None
                elif isinstance(result.optimize_result.list[0][key], np.ndarray):
>                   np.testing.assert_almost_equal(
                        result.optimize_result.list[0][key],
                        result_from_hdf5.optimize_result.list[0][key]
                    )
E                   AssertionError: 
E                   Arrays are not almost equal to 7 decimals
E                   
E                   Mismatched elements: 2 / 2 (100%)
E                   Max absolute difference: 0.0001968
E                   Max relative difference: 0.00014251
E                    x: array([-0.915745 , -7.9470828])
E                    y: array([-0.9156145, -7.946886 ])

test/base/test_store.py:424: AssertionError

[PaulJonasJost]

mhh normally just running it again solves the problem... I think that also sometimes happens with other hdf5 tests. Not exactly sure why this happens tbh

[dweindl]

mhh normally just running it again solves the problem

That's not how a test suite should behave...

[FFroehlich]

mhh normally just running it again solves the problem

That's not how a test suite should behave...

Especially since writing to and reading from hdf5 should roundtrip without loss of information and there is no reason to expect any kind of stochasticity 🤔

[dweindl]

The reported values suggest that this is related to

WARNING pypesto.optimize.optimizer:optimizer.py:170 Parameters obtained from history and optimizer do not match: [-0.91561453 -7.946886 ], [-0.91574501 -7.9470828 ]

see also https://github.com/ICB-DCM/pyPESTO/issues/327

[yannikschaelte]

I hope this has been fixed in 0.2.11. Something weird may or may not have remained, so let's keep an eye on whether tests fail again.

[dweindl]

Something weird may or may not have remained, so let's keep an eye on whether tests fail again.

Something weird has remained...

e.g. https://github.com/ICB-DCM/pyPESTO/runs/5743093422?check_suite_focus=true:

________________________ test_result_from_hdf5_history _________________________

hdf5_file = '/tmp/tmpc_94gg6k/file.hdf5'

    def test_result_from_hdf5_history(hdf5_file):
        problem = create_petab_problem()
    
        history_options_hdf5 = pypesto.HistoryOptions(
            trace_record=True,
            storage_file=hdf5_file,
        )
        # optimize with history saved to hdf5
        result = optimize.minimize(
            problem=problem,
            n_starts=1,
            history_options=history_options_hdf5,
            progress_bar=False,
        )
    
        result_from_hdf5 = optimization_result_from_history(
            filename=hdf5_file, problem=problem
        )
    
        # Currently 'exitflag', 'time' and 'message' are not loaded.
        arguments = [
            ID,
            X,
            FVAL,
            GRAD,
            HESS,
            RES,
            SRES,
            N_FVAL,
            N_GRAD,
            N_HESS,
            N_RES,
            N_SRES,
            X0,
            FVAL0,
        ]
        for key in arguments:
            if result.optimize_result.list[0][key] is None:
                assert result_from_hdf5.optimize_result.list[0][key] is None
            elif isinstance(result.optimize_result.list[0][key], np.ndarray):
>               assert np.allclose(
                    result.optimize_result.list[0][key],
                    result_from_hdf5.optimize_result.list[0][key],
                ), key
E               AssertionError: grad
E               assert False
E                +  where False = <function allclose at 0x7f0dce8875e0>(array([ 0.01186742, -0.01155628]), array([ 0.01136439, -0.01128129]))
E                +    where <function allclose at 0x7f0dce8875e0> = np.allclose

test/base/test_store.py:524: AssertionError

[dweindl]

And still... https://github.com/ICB-DCM/pyPESTO/runs/5881283280?check_suite_focus=true

[dweindl]

Should be fixed, i think.