Added an option of chunk size to the relevant functions. Not outsourced as a function yet, if wanted would put it into a AmiciCalculator function.

Also: I did not yet change the amici_predictor.

Would be happy for feedback :)

Mar 26 '21 16:03 PaulJonasJost

So, how do we proceed here? Current status is:

The way AMICI simulation is run in pyPESTO (not chunked) leads to crashing (out of memory) of certain optimization when dealing with large models. So, these models can't be simulated/optimized with pyPESTO at all at the moment
The chunking feature, which is used in the predictor functionality resolves this problem
Therefore, our plan was to implement this feature also in the objective evaluation of pyPESTO.

Sure, we can also try to resolve that in AMICI, but even doing it in a double way is no harm. We're talking here about 50-100 lines of code in python, which will help the situation, and which will be definitely more flexible to adapt than anything we do in AMICI itself. Optimizing memory handling in AMICI is always good, but I'd do that on top...

What's your opinion on that @FFroehlich, @yannikschaelte, @dweindl ?

Apr 06 '21 07:04 paulstapor

For me, all sounds fine. Should be not too much overhead, so fine to implement twice.

Apr 06 '21 07:04 yannikschaelte

I don't understand the issue or question. Improved memory handling is already implemented in AMICI and pypesto should make use of that. I don't see any issues adding additional code in pypesto.

Apr 06 '21 17:04 FFroehlich

The issue is that even though amici_solver.setReturnDataReportingMode(2) was used, evaluation of the objective failed in some cases (@LeonardSchmiester and @PaulJonasJost had this problem) due to going out of memory. In contrast, the implementation in the pypesto predictions worked, since it uses chunking. @LeonardSchmiester and @PaulJonasJost will retry with Amici 0.11.15 and again try to set the return data reporting explicitly before calling the objective. However, if this doesn't resolve the problem, then also no automatic handling within pyPESTO of the reporting mode will solve it.

Apr 07 '21 10:04 paulstapor

Still don't understand the problem. If additional engineering in pypesto helps, go for it. Usually more sustainable to find the actual issue and fix it where it occurs though. I guess one of the issues could be the computation of the FIM, which currently will only be turned off when using adjoint sensitivities (I think?).

Apr 07 '21 14:04 FFroehlich

Just tried it again with Amici 0.11.15 and still get SIGKILLed

Apr 07 '21 14:04 PaulJonasJost

Is there an example where I can try to reproduce this?

Apr 07 '21 17:04 FFroehlich

Pretty much what was said already. Should definitely make use of AMICI amici_solver.setReturnDataReportingMode (I thought I was told this was already the case). I think this will be helpful in more cases than chunking.

Chunking will probably also be helpful and it's good to have it in addition. I would default to returning the same items as before, but allow passing a callback function for filtering the results.

Apr 12 '21 08:04 dweindl

Should be reproducible with the large-scale pan-cancer model. So e.g. https://github.com/ICB-DCM/CS_Signalling_ERBB_RAS_AKT/tree/master/FroehlichKes2018/PEtab

I also thought this should already work. However, it did not resolve the memory issues I had.

Apr 12 '21 08:04 LeonardSchmiester

I can't reproduce any SIGKILL for that model. Using https://github.com/AMICI-dev/AMICI/blob/master/tests/performance/test.py for import/compilation/simulation and amici 0.11.16.

Apr 14 '21 15:04 FFroehlich

How many conditions did you simulate at a time? The performance test only simulates for one condition, as far as I'm aware of. And the problem only arises when simulating several dozens/hundreds (depending on the computer...)

Apr 15 '21 14:04 paulstapor

I am confused. I thought the issue was that using amici_solver.setReturnDataReportingMode leads to a sigkill?

Apr 15 '21 16:04 FFroehlich

Yes, but it does so only if you simulate many conditions, say all 9k from that PEtab model via something like pypesto's objective....

Apr 15 '21 17:04 paulstapor

That sounds like a memory issue though, where chunking may help. Not sure what we are discussing about at this point ...

Apr 15 '21 18:04 FFroehlich

I am also not sure what exactly we are now discussing anymore. Would like to ask whether I should then delete the branch or whether (and in which direction) I should proceed?

Apr 30 '21 15:04 PaulJonasJost

Codecov Report

@@             Coverage Diff             @@
##           develop     #608      +/-   ##
===========================================
- Coverage    34.39%   34.30%   -0.09%     
===========================================
  Files           96       96              
  Lines         6513     6524      +11     
===========================================
- Hits          2240     2238       -2     
- Misses        4273     4286      +13

Impacted Files	Coverage Δ
pypesto/objective/amici.py	`19.71% <ø> (ø)`
pypesto/objective/amici_calculator.py	`22.38% <0.00%> (-2.62%)`	:arrow_down:
pypesto/objective/amici_util.py	`20.33% <0.00%> (-0.72%)`	:arrow_down:
pypesto/engine/multi_process.py	`92.59% <0.00%> (-7.41%)`	:arrow_down:

Continue to review full report at Codecov.

Apr 30 '21 22:04 codecov-commenter

Feature simulation chunks

Codecov Report