dask
Allow usage of user-defined script
This PR makes initializing the DRMAACluster object with a user-defined script possible. Trivial fix, the script attribute was not set before using it when intializing the template.
I can see that this is a duplicate of #22 or #24. Can any of these 3 be merged - happy to defer obviously.
It would be great to understand more your use case for wanting a custom script and whether improvements in dask-drmaa could alleviate that or not.
A user-defined script would allow setting runtime parameters that are important in HPC environments (e.g. the path of the script (currently hardcoded, most of the time needed to be on a shared filesystem), memory, max runtime, etc). Regardless, it's a one line bug fix that will restore the flexibility of the API at probably the originally intended level, so why not just merge it (or #22 or #24, whatever works as long as it's fixed).
@h0bot merging PRs without tests can introduce significant costs for maintainers down the line, which can be problematic for volunteer-run projects. If you're able to either add an appropriate test to #22 or identify and fix the cause of the failure in #24 then that is probably the best way forward here. I appreciate your engagement here.
To be clear, I'm not against adding this functionality somehow (especially if it is useful to people), @h0bot. The number of issues and PRs built up in this direction indicate this is important to people. However @mrocklin is right. It really needs at lease one useful, working test. That's as much for keeping maintenance reasonable on our end as it is about trying to guarantee you a certain level of functionality going forward (without accidentally breaking it).
Independently though I think we could push a lot of the stuff you have described through DRMAA's nativeSpecification parameter, which is just a catch all string that gets forwarded verbatim to whatever submission command you are using (e.g. qsub on SGE). Given getting the scripting functionality tested has been hard, maybe this would be an easier path forward. We could even look at using nativeSpecification in combination with some knowledge about the scheduler type to add in info like resource limits in a more user-friendly way. ( https://github.com/dask/dask-drmaa/issues/66 ) Hence the question. 😉