Optional steps and/or queries

[andrioni]

The ability to skip some steps and/or queries sometimes would be nice.

My current use case: I'm using sql-runner to load data into Redshift using a dynamically generated manifest file, and sometimes the manifest file ends up empty. Whenever that happens, COPY just errors out.

Another option would be to say certain error codes are acceptable.

(this also might move sql-runner dangerously close to being a full-fledged workflow scheduler, heh)

Related to #32.

[alexanderdean]

Interesting... from an ops perspective I'm not a huge fan of DAGs with conditional steps - they tend to encourage silent failures, where everything is operating "normally" but the end goal of the DAG is not being achieved (Q: "why is there no data in my table?" A: "not sure, I'm not seeing any jobflow failures being reported")...

[iaingray]

This would be useful to us as well to provide graceful failure. At the moment we have situations where (for example) reference table update queries break for some reason, and as a result the entire ETL process dies and has to be restarted from the point of breakage.

I agree though it shouldn't be silent, some sort of alert needs to be fired off to let ops know a step has failed.

[alexanderdean]

If you have this DAG:

Script A > Script B > Script C

and you are happy to keep processing Script C after Script B fails, then that implies to me that Script C does not depend on Script B, so your DAG is wrong, and you should either have:

Script A > Script C > Script B

or

Script A > Script C
         > Script B

or even two separate DAGs. What am I missing?