[FEATURE]: Allow parameterization of generic/template buildspecs

[wspear]

To support the creation and maintenance of multiple very similar tests, I suggest implementing buildspec templates which allow changes to the buildspec contents using input parameters. They might look something like this:

buildspecs:
  <TEST>_e4stestsuite_e4s_21.05:
    type: script
    executor: cori.slurm.haswell_premium
    description: Run <TEST> from E4S Testsuite for e4s/21.05
    tags: [e4s]
    sbatch: ["-t 30", "-N 1"]
    run: |
      module swap intel intel/19.1.3.304
      module load e4s/21.05
      git clone https://github.com/E4S-Project/testsuite
      cd testsuite
      sh test-all.sh --color-off validation_tests/<TEST> --print-logs --settings settings.cori.sh

And could be run like this: buildtest build -b e4s_testsuite.yml --parameters "TEST=adios2;PARAMETER2=another-parameter" So a dictionary type argument is passed to the build command and string substitution takes place before the template spec is interpreted.

Considerations:

• It would be preferable to keep template specs valid yaml files. That might require extending the schema so only values need to be parameterized (i.e. adding a name: "string" field to buildspec)
• A basic implementation of this could work by setting environment variables for the test's run environment. This approach could control the contents of the run script, but other elements (test name and description) would have to be set separately.
• A method of running a test template multiple times iterating over a list of input parameters is necessary to employ this feature effectively in CI.
• I recommend a hard distinction between full buildspecs and template specs. Running a regular buildspec with parametric input results in a failure, running a template without parametric input results in a failure. There should be filename conventions to distinguish templates from regular buildspecs.

Post question in Slack

• [X] I agree that I posted my question in slack before creating this issue

Is there an existing issue

• [X] I confirm there is no existing issue for this issue

[etpalmer63]

:+1:

I think we could use this for the multiple changing E4S versions. For example, if we could pass "E4S_VERSION=23.05" into each buildspec, we wouldn't need to edit them each time we changed versions.

[shahzebsiddiqui]

I think if we have a variable declaration like this when running buildtest build.

E4S_VERSION=23.05 buildtest build -t e4s

Then have a buildspec something like as follows where variable is referenced in script it should technically get the value from the shell and just inject it in. It shouldnt require any changes to code. It would make the buildspec more flexible. It would be worth a try and see if it works.

buildspecs:
  trilinos_e4s_testsuite_22.11:
    type: script
    executor: perlmutter.slurm.regular
    description: Run E4S Testsuite trilinos test for e4s/22.11 stack
    tags: [e4s]
    sbatch: ["-t 30", "-N 1", "-G 1", "-A m3503_g", "-C gpu"]
    run: |
      module load e4s/$E4S_VERSION
      spack env activate -V gcc
      spack load --first cmake
      git clone https://github.com/E4S-Project/testsuite
      cd testsuite
      sh test-all.sh --color-off validation_tests/trilinos --print-logs --settings settings.cray.sh
maintainers:
  - shahzebsiddiqui
  - wspear

The feature for test parameterization is still valid but this would be a workaround