Pablo Selem
Pablo Selem
Alternatively so that we can't see the shared keys in clear text, but we still have them on node, can we simply re-generate a SAS every time?
We will need to update some logic here. Currently this is downloaded via the start task's resource file list. It will need to be downloaded as part of the body...
Hi @emlyn - overall this change makes a lot of sense to me. The only things that I'm not 100% sold on yet are how you're printing off the logs...
Thanks @emlyn! Let us know once you've got the next PR ready to view @jafreck do we have an issue to track viewing/downloading logs to a directory?
Do we know what the impact of that would be on the spark scheduler? Probably nothing at this point, but I'm not sure about the value of increasing it since...
Possibly worth investigating azure files for this. What all is stored in the metastore_db?
I'm not sure I like having this at the root since it could potentially just spew a bunch of stuff out there. Having it in a known location seems better...
Based on the definition of WORKDIR(https://docs.docker.com/engine/reference/builder/#workdir) it will only apply to the very last command in the container right? Since the last command is running Spark I think $SPARK_HOME does...
@dustindall - Can you provide a bit more info around your scenario? Do you just want to create a job w/ Wait = FALSE and a schedule to run it...
Actually, the Batch scheduler should be able to do both one-offs and routine jobs. It would be reasonably easy to integrate this into the package if we used a pure...