Empty loss stream cause run_ktools to hang
Issue Description
Occasionally run_ktools.sh hangs when there is one or more empty pipes. The cause appears to be when summarycalc receives an empty loss stream. It is an intermittent problem, sometimes the script completes successfully even if some pipes loss streams are empty.
This is more likely to happen;
- When there are more processes than events in the model run (e.g. a multi-core machine with a single event model)
- When there are many events that do not cause a loss to the portfolio and they all coincide in a single pipe
Part of the solution is setting the number of parallel processes appropriately for the run. However, there will still be plausible cases where no losses are generated in a whole pipe (the portfolio is very small, or low risk, or there is a high loss threshold applied to losses). This bug needs to be fixed so that the script always completes, even though some pipes may not contribute any losses to outputs.
Steps to Reproduce (Bugs only)
- Simple script and data package for Paris Windstorm
Version / Environment information
Ubuntu 20 Observed both natively and inside model execution worker docker container Observed in latest ktools and 6 month old version
Example data / logs
Example from @benhayes21 test_named_pipe_fail.tar.gz
to test
@johcarter - I tested this with 1.23.14 today with PiWind using 2 events and 8 processors and the issues doesn't seem to be there
The example script above still hangs - though I'm not sure what it's supposed to be doing?