Support retry reliable stage execution.

[zootalures]

Stages/ functions can fail or timeout.

If I produce a stage that is idempotent I want for the platform to support retrying of that stage on my behalf.

Questions:

• How much control to expose via API
• How to express differing retry behaviours in code.

Relates to:

• could be implemented within the flow-service or perhaps better with fn itself by expanding the behaviour of async retries.
• https://github.com/fnproject/flow/issues/86

[jan-g]

Just a note - during load testing I'm seeing this (occasionally) during the graph termination stages; if Fn is maxed out then the terminator (which I'd have no issue with declaring "should be idempotent" in particular) oughtta be retried ratehr than dropped on the floor - it's our last and best opportunity as a flow author to clean up.