sos
sos copied to clipboard
Remote checks on dependent steps not working
Hello Bo, thanks again for providing this excellent tool. I have a workflow where I build a custom database for kraken and then have a subsequent step that uses kraken and the database on some samples with a dependency on the database build step. However, when I change the database build step, the subsequent steps do not re-run. Here's an example.
%sosrun depends_test -q yale_hpc_task_spooler \
-r yale_hpc_task_spooler mem="1G" cores=1 walltime="00:05:00" nodes=1
[touch_files]
output: f'/home/pgc29/scratch60/test.txt'
task: walltime='00:01:00', mem='1G', cores=1, nodes=1,
workdir='/home/pgc29/scratch60/'
run: expand=True
touch {_output}
[depends_test]
depends: sos_step('touch_files')
output: f'/home/pgc29/data/test2.txt'
task: walltime='00:01:00', mem='1G', cores=1, nodes=1,
workdir='/home/pgc29/scratch60/'
run: expand=True
touch {_output}
But if I then modify the first step to
[touch_files]
output: f'/home/pgc29/scratch60//test.txt'
task: walltime='00:01:00', mem='1G', cores=1, nodes=1,
workdir='/home/pgc29/scratch60/'
run: expand=True
echo "test" > {_output}
And run depends_test again, it re-runs touch_files but skips depends_test
INFO: Running touch_files:
INFO: tce6f8d6f7553ab25 submitted to yale_hpc_task_spooler with job id 30325020
INFO: Waiting for the completion of 1 task.
INFO: tce6f8d6f7553ab25 received '/home/pgc29/scratch60/test.txt' from yale_hpc_task_spooler
INFO: touch_files output: /home/pgc29/scratch60/test.txt
INFO: Running depends_test:
INFO: depends_test (index=0) is ignored due to saved signature
INFO: depends_test (index=0) is ignored due to saved signature
INFO: depends_test output: /home/pgc29/scratch60/test2.txt
INFO: Workflow depends_test (ID=w01db23b738dc700b) is executed successfully with 1 completed step, 1 ignored step and 1 completed task.
It is a moderately complex setup. Let me try to reproduce it and see why signature does not work.