singularityhub
Using the sed command in Singularity Exec does not work. Similar with Redirect ">" symbol.
Command line works as expected:
singularity exec --writable-tmpfs instance://reclusive_cat_2780 sed -i 's/REPLACE_THIS_LINE/ <parameter name="Raw data file names"><file>\/Work\/III_mzML\/Dorrestein_GnPS_P14_G10_7V_Rep01_Neg_29Dec20_Fiji_Infusion_Min50_MA-csum-Min20_3.mzML<\/file><\/parameter>/' /Work/MZmine_FeatureFinder-batch.xml
Python Command doesn't work:
file_name = "test_file"
command_list_1 = """sed -i \'s/REPLACE_THIS_LINE/ <parameter name="Raw data file names"><file>\/Work\/III_mzML\/""" +file_name + """<\/file><\/parameter>/\' /Work/MZmine_FeatureFinder-batch.xml"""`
Client.execute(myinstance,command_list_1, options=['--writable-tmpfs'],quiet=False)`
Error message: /bin/sed: -e expression #1, char 1: unknown command: `''
Try putting the sed command in a bash script and running that. We split the woman’s with shlex and your sed FU probably doesn’t work well with that approach.
Just wrote a quick python script to do it instead. Still would be a great thing to fix for users. I’m converting a docker pipeline to singularity for HPC and took quite a while to track down exactly where it was failing.
Try running the command with subprocess as spython does and you will likely run into the same issue.
spython isn’t a workflow tool - it’s a thin wrapper to running singularity. Likely a workflow tool would take a command and instead write to a temporary file first, but I consider that out of scope here because it’s not up to spython to make that decision for the user.
Okay, your call of course! Just was surprised that the wrapper has a bit less functionality than the command line interface because it seems pretty well supported. Could be helpful to note in documentation for future users.
The wrapper uses subprocess, and any tool that does similar will have similar issue with things like pipe or sed. If you’d like to do a PR to update docs with a suggestion I’d be happy to review that. I do not see singularity Python as being responsible for managing a workflow for which it’s really better practice to write complex commands to a bash script so they are properly run with a shell.