No way to control working directory on remote host when running commands via luigi.contrib.ssh.RemoteContext

[ThePletch]

When running RemoteContext('host').Popen(['run', 'my_command'], cwd='/some/directory'), I would expect the cwd param passed to Popen to control the working directory from which the command is run on the remote host, but in its current incarnation, it only controls the working directory from which the ssh command is run.

Attempting to pass in a cd command in addition to the command, due to the way Luigi structures its SSH invocation, causes the command to cd to the specified directory, close the ssh session, and invoke the second command on the executing host. Example as follows:

RemoteContext('host').Popen(['cd', '/some/directory', '&&', 'run', 'my_command'])
# => becomes 'ssh host cd /some/directory && run my_command'
#     which runs 'ssh host cd /some/directory', then separately runs 'run my_command'

The only way I can find to control the working directory on the remote host on a command-by-command basis is to convert the command I'd like to execute to a string and append a cd command to it, like so:

RemoteContext('host').Popen(['cd /some/directory && run my_command'])

This, however, means I need to manage escaping of special characters myself and deal with all the other corner cases of running and formatting shell commands. It's also nearly impossible to arrive at this solution without delving into the Luigi source code and seeing exactly how RemoteContext works internally, since this requires following a hybrid of the two patterns Python has established for specifying commands to invoke.

RemoteContext should be updated to either propagate the cwd setting to the remote host or have a separate argument that lets you specify the remote working directory.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If closed, you may revisit when your time allows and reopen! Thank you for your contributions.