isce2 icon indicating copy to clipboard operation
isce2 copied to clipboard
verified publisher icon isce-framework

stackSentinel background task fix

Open hardreddata opened this issue 4 years ago • 6 comments

There is a problem with https://github.com/isce-framework/isce2/issues/223 in which any tasks running in the background will continue in the background when the next step of the process begins (eg: run_03 can bleed into run_04). This causes issues when some steps try and use incomplete data.

The proposed solution writes a final wait in any shell scripts which leverage num_process.

I welcome any feedback.

hardreddata avatar Apr 14 '21 05:04 hardreddata

@yunjunz I would be happy to change

        line_cnt = 1
        configObj.write_wrapper_config2run_file(configName, line_cnt)

to

        configObj.write_wrapper_config2run_file(configName)

in the three occurrences. I left as above as it might be simpler to follow.

Presently the stackSentinel.py num_process is buggy so I think good to get this merged. Apologies for the delay.

hardreddata avatar Jul 22 '22 03:07 hardreddata

Thank you for getting back to this @RussellGrew. Could you do the following?

  1. Rebase this PR against the latest main branch, as the stack processor has been updated and GitHub flagged conflicts.
  2. Apply the same enumerate style call to the new functions added within the class run(object).

@yunjunz I would be happy to change

        line_cnt = 1
        configObj.write_wrapper_config2run_file(configName, line_cnt)

to

        configObj.write_wrapper_config2run_file(configName)

in the three occurrences. I left as above as it might be simpler to follow.

I would prefer the latter one. We don't need to show the details if we don't have to.

yunjunz avatar Jul 26 '22 06:07 yunjunz

@yunjunz I think that is everything.

hardreddata avatar Jul 27 '22 04:07 hardreddata

These two still stand I believe:

  1. Rebase this PR against the latest main branch, as the stack processor has been updated and GitHub flagged conflicts.
  2. Apply the same enumerate style call to the new functions added within the class run(object). E.g., filtIon(). They will be visible in this PR after rebasing.

yunjunz avatar Jul 27 '22 05:07 yunjunz

@yunjunz see https://github.com/isce-framework/isce2/pull/271/commits/4ffbffff0b269d13e53775394cb65da4278a1b6f and https://github.com/isce-framework/isce2/pull/271/commits/8971f5292f5803b036ac3b14e5a3893d842a2bac

Though this was the first time I have attempted to update a branch for an in progress PR. Hopefully it went ok.

Edit - I see that it did not. Leave it with me.

hardreddata avatar Jul 27 '22 05:07 hardreddata

That should be much closer.

You did this to yourself with your notes of encouragement on https://mintpy.readthedocs.io/en/latest/#5_contributing

I see the CI has now passed.

I am not familiar with these new ionosphere steps. I checked that the config is written out, and that step 17 and 18 which consider the numProcess are written correctly.

Cheers.

hardreddata avatar Jul 27 '22 06:07 hardreddata