merlin icon indicating copy to clipboard operation
merlin copied to clipboard
verified publisher icon LLNL

Add support for non-merlin blocks in specification file

Open jimagaffney opened this issue 3 years ago • 3 comments

Adding capability for non-user blocks of yaml code to be included in the merlin specification, and propegated to the .partial and .expanded versions at runtime. This allows the users to define yaml specifications for steps in the workflow in the same file, and for user code to perform consistency checks between the merlin spec and code in the steps - for example, checking consistency between variable names, etc.

A new "user" block has been added to the MerlinSpec class in spec/specification.py. The user block is not included in any checking and does not have default values. The yaml code has shell and merlin parameters substituted and is copied to the working directories verbatim.

Changelog:

  • Added a static method load_user_block to the MerlinSpec class, which loads the user block from the yaml specification and returns the python representation
  • In functions load_specification and load_spec_from_string, added definitions of self.user through calls to load_user_block
  • In the yaml_sections and sections properties, added references to self.user
  • In the dict_to_string method of MerlinSpec object, add user block support for pretty printing of lists

When no user block is present in the yaml file, the MerlinSpec.user attribute is an empty dictionary

Test Functionality:

  • correct generation of problems in merlin test problems (with -clean option)
  • correct generation of problems and variable substitution when a user block is added to the test problems (with -clean option)
  • complete execution of merlin test problems

jimagaffney avatar Jun 14 '22 21:06 jimagaffney

Please change PR to go into develop, not main

—— J. Luc Peterson, PhD (he/him) LLNL @.***

On Jun 14, 2022, at 4:52 PM, Jim Gaffney @.***> wrote:



You can view, comment on, or merge this pull request online at:

https://github.com/LLNL/merlin/pull/367https://urldefense.us/v3/__https://github.com/LLNL/merlin/pull/367__;!!G2kpM7uM-TzIFchu!i644q1r4nZzXEjKdJMBhHmGKtg1ACsfnbyzV5clipOtdI1UH1yQc-DhtVdzO8DQULwQ$

Commit Summary

File Changes

(1 filehttps://urldefense.us/v3/__https://github.com/LLNL/merlin/pull/367/files__;!!G2kpM7uM-TzIFchu!i644q1r4nZzXEjKdJMBhHmGKtg1ACsfnbyzV5clipOtdI1UH1yQc-DhtVdzO6ZdA1EI$)

Patch Links:

— Reply to this email directly, view it on GitHubhttps://urldefense.us/v3/__https://github.com/LLNL/merlin/pull/367__;!!G2kpM7uM-TzIFchu!i644q1r4nZzXEjKdJMBhHmGKtg1ACsfnbyzV5clipOtdI1UH1yQc-DhtVdzO8DQULwQ$, or unsubscribehttps://urldefense.us/v3/__https://github.com/notifications/unsubscribe-auth/AAEPYRHL4YYTPVGI5OECE4TVPD5I5ANCNFSM5YZFTDGQ__;!!G2kpM7uM-TzIFchu!i644q1r4nZzXEjKdJMBhHmGKtg1ACsfnbyzV5clipOtdI1UH1yQc-DhtVdzO8GoU30Q$. You are receiving this because you are subscribed to this thread.Message ID: @.***>

lucpeterson avatar Jun 15 '22 00:06 lucpeterson

OK done

jimagaffney avatar Jun 15 '22 00:06 jimagaffney

Is there an integration test we can add to make sure it’s working?

lucpeterson avatar Jun 16 '22 21:06 lucpeterson

Resolved with PR #376

ryannova avatar Oct 11 '22 00:10 ryannova