Need a way to terminate the stuck task

[xavierchow]

Sometimes the task keeps running for a long time and gets stuck, need a way to stop it.

[JuhaS]

First step would be to add timeout to bash_executor.py.

There is _timeout in the sh package that we use to run the commands: https://amoffat.github.io/sh/sections/special_arguments.html?highlight=timeout#timeout

@zbal looks ok to you? Ok to send SIGKILL on timeout?

We should also add timeout handling on all the other executors.

Other option is to set up timeout on pipeline execution level, would require to run the execution in a thread and then attempt to kill it after the timeout from the main thread.

[JuhaS]

I implemented the _timeout and set it to 1 hour for bash tasks. This is at least a partial solution for this issue.