pecan icon indicating copy to clipboard operation
pecan copied to clipboard

Improved the current SDA workflow to reach the North American runs with 6400 sites.

Open DongchenZ opened this issue 1 year ago • 0 comments

Description

Motivation and Context

  1. The foreach package seems to be better compared to the furrr package concerning memory allocation. Thus, in this PR, I replaced every furrr with foreach during the general SDA workflow.
  2. The computational power and memory are limited when executing certain SDA procedures locally (e.g., splitting meteorology files, writing configuration files, reading SDA outputs, running Bayesian MCMC analysis part, and removing files (e.g., removing NC files after the first SDA run)). Therefore, in the PR, I developed features of qsub submissions (specified by the batch.settings section in the XML file, which is also documented in the pecan book) during the SDA workflow.
  3. To avoid the complex if-else usage within the current sda.enkf_Multisite function for the above batch job submissions, I developed a new sda.enkf_NorthAmerica function (although most of them are just copy-and-paste from the sda.enkf_Multisite function), which is cleaner and will only be used if the batch.settings is not empty.

Review Time Estimate

  • [ ] Immediately
  • [ ] Within one week
  • [x] When possible

Types of changes

  • [x] Bug fix (non-breaking change which fixes an issue)
  • [x] New feature (non-breaking change which adds functionality)
  • [ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • [ ] My change requires a change to the documentation.
  • [ ] My name is in the list of CITATION.cff
  • [ ] I have updated the CHANGELOG.md.
  • [ ] I have updated the documentation accordingly.
  • [ ] I have read the CONTRIBUTING document.
  • [ ] I have added tests to cover my changes.
  • [ ] All new and existing tests passed.

DongchenZ avatar Jul 22 '24 22:07 DongchenZ