Bootstrap install script

[jerowe]

Ping @boegel - how is it going?

I've had this kicking around for awhile as an internal resource, and wanted to share it with you. I am pretty bad at bash scripting, so let me know if there is anything you think should be changed.

Its very simple - it just bootstraps an easybuild with an optional lmod or environmental variables install using anaconda. It first downloads the anaconda script and installs it to a location specified by --easybuild_base in the script.

https://gist.github.com/jerowe/20caccac532355dba2288b6fb665d52b

Get help

➜  infrastructure ./deploy_gencore_modules.sh  -h                                   
EasyBuild Bootstrap: option requires an argument -- 'h'
EasyBuild Bootstrap - Bootstrap method of installing easybuild
Usage example:
easybuild-bootstrap (-e|--easybuild_base) string (-p|--easybuild_prefix) char [(-h|--help) boolean] [(-E|--environment_modules)] [(-L|--lmod)]
Options:
-h or --help boolean: Displays this information.
-E or --environment_modules: Include Environment Modules.
-e or --easybuild_base string: Easybuild software base. Required.
-p or --easybuild_prefix char: Easybuild Prefix - Base for software and modules created by easybuild. Required.
-L or --lmod: Include Lmod.

Install with lmod

./deploy_eb_modules.sh  -e "${HOME}/.eb" -p "${HOME}/.eb" -L

This is not a real issue, so please feel free to close it out, unless there is some more functionality you suggest I add.

[boegel]

@jerowe This is interesting, thanks for sharing!

Maybe you should make a PR for this in the easybuid/scripts directory at https://github.com/hpcugent/easybuild-framework, together with a small README to explain what it does and how to use it?

I gave it a quick try, but couldn't figure out what I'm doing wrong here:

$ ./easybuild_bootstrap_install.sh -e $PWD

easybuild_base is required

Also, is there a way to avoid hardcoding versions?

Style-wise, I'd suggest renaming --easybuild_base to --easybuild-base, I rarely see underscores being used in option names.

[jerowe]

I'm sure there is a way to avoid hardcoding versions. They should be an option with the default set to the values I have in there. For now those are the only versions though. ;-)

It seems like the -e $PWD should have worked, or at least told you to supply the easybuild_prefix (corresponds to EASYBUILD_PREFIX) instead of the -e.

➜  infrastructure ./deploy_gencore_modules.sh -e "THING"                         
easybuild_prefix is required

➜  infrastructure ./deploy_gencore_modules.sh -e "$PWD"                         
easybuild_prefix is required

I get easybuild_prefix is required, not base.

Words just cannot express how bad I am at bash scripting though, so let me run through with some more tests. I can handle bats tests for bash scripts and that's about it.

[jerowe]

Ah -

easybuild_base is the place where easybuild is actually installed to. easybuild_prefix corresponds to the EASYBUILD_PREFIX variable. I have them separate, because I like them that way, but maybe they shouldn't be?

[boegel]

Hmm, I keep getting the same error, maybe it's because of a subtle change in bash on macOS?

$ echo $PWD
/tmp/jerowe

$ ./easybuild_bootstrap_install.sh -e "$PWD" -p $"PWD"

easybuild_base is required
...

Supporting to install EasyBuild and the prefix for EasyBuild to use differently makes sense, but I'm not sure it's worth the trouble, typically you'd only use on location.

[jerowe]

Could be, but it would be nice if it was also supported on mac.

Let me take a look.

[jerowe]

Doesn't work for my officemate who is using macosx.

sighs

[jerowe]

How do I check that a bash variable is defined on osx bash?

[boegel]

-z seems to work...

$ export JEROWE=1; test -z $JEROWE; echo $?
1
$ unset JEROWE
$ test -z $JEROWE; echo $?
0

[boegel]

Maybe it's because you're not using [[ ... ]]:

$ if [[ ! -z "$JEROWE" ]]; then echo $JEROWE; else echo "not set"; fi
not set
$ export JEROWE=foo
$ if [[ ! -z "$JEROWE" ]]; then echo $JEROWE; else echo "not set"; fi
foo

[jerowe]

Want to check it in the script? I think my office mate has had enough. ;-)

btw, I used http://getoptgenerator.dafuer.es/ <- this to generate these.

[boegel]

well, I think if [[ ! -z "$easybuild_base" ]] is a better way of checking it, but I don't know the details from the top of my head either...

[jerowe]

Well, the basic fuctionality doesn't work, but I added default version variables for eb, lmod, and environment modules!

[jerowe]

I think -z checks if a variable is undefined or unset, but maybe the double brackets will work?

[boegel]

@jerowe Yes, that's important I think, see https://stackoverflow.com/questions/669452/is-preferable-over-in-bash

[jerowe]

I don't like asking you to be a guinea pig, but I don't have access to a mac. I updated the gist. Could you check it?

https://gist.github.com/jerowe/20caccac532355dba2288b6fb665d52b

[jerowe]

I'm happy to open a PR, but I need to reformat my computer so I want to wait a bit. ;-)

If you don't pinging me when there is a new easybuild release I don't mind updating the packages in conda.

[boegel]

$ md5 easybuild_bootstrap_install.sh
MD5 (easybuild_bootstrap_install.sh) = 553bdc6d6cdde9c91312511d735bd4a3
$ ./easybuild_bootstrap_install.sh -e "$PWD" -p $"PWD"
easybuild_base is required

Still the same problem, weird...

I hope to issue a new EasyBuild release (v3.3.0) next week. You should get on the EasyBuild mailing list, and set up a filter for [ANN] in the subject. Or keep an eye on our Twitter account.

[jerowe]

This is really strange. So on linux its "-z $THING" and on mac its "! -z $THING"

Is that what I'm getting?

[jerowe]

Maybe have two separate scripts? One for OSX and one for Linux?

[boegel]

No, the behaviour of -z is exactly the same in bash on macOS or Linux (it's still bash...).

I don't know what's going wrong exactly, but a separate script for Linux vs macOS doesn't make sense to me.

[jerowe]

@boegel, I think I got it this time! Could you try it out?

[jerowe]

The bit at the bottom is wrong, but at least the initial command line parsing is (hopefully) correct.

[boegel]

Hmm... Still the same issue?

$ md5 ~/work/WIP/easybuild_bootstrap_install.sh
MD5 (/Users/kehoste/work/WIP/easybuild_bootstrap_install.sh) = 625e330f4ed0f67366c9ab8e58bcbf73
$ ~/work/WIP/easybuild_bootstrap_install.sh -e $PWD
easybuild_base is required
EasyBuild Bootstrap - Bootstrap method of installing easybuild
Usage example:
easybuild-bootstrap (-e|--easybuild_install_base) char [(-h|--help) boolean] [(-E|--environment_modules)] [(-L|--lmod)] [(-b|--eb_version) char] [(-l|--lmod_version) char] [(-m|--em_version) char]
Options:
-h or --help boolean: Displays this information.
-E or --environment_modules: Include Environment Modules.
-e or --easybuild_install_base char: Easybuild software base. Required.
-L or --lmod: Include Lmod.
-b or --eb_version char: Easybuild Version  - Default 3.1.0.
-l or --lmod_version char: Lmod Version - Default 7.3.16.
-m or --em_version char: Environment Modules Version - Default 3.2.10.

[jerowe]

I think I created new issues. ;-)

On Mon, Aug 28, 2017 at 1:38 PM Kenneth Hoste [email protected] wrote:

Hmm... Still the same issue?

$ md5 ~/work/WIP/easybuild_bootstrap_install.sh MD5 (/Users/kehoste/work/WIP/easybuild_bootstrap_install.sh) = 625e330f4ed0f67366c9ab8e58bcbf73 $ ~/work/WIP/easybuild_bootstrap_install.sh -e $PWD easybuild_base is required EasyBuild Bootstrap - Bootstrap method of installing easybuild Usage example: easybuild-bootstrap (-e|--easybuild_install_base) char [(-h|--help) boolean] [(-E|--environment_modules)] [(-L|--lmod)] [(-b|--eb_version) char] [(-l|--lmod_version) char] [(-m|--em_version) char] Options: -h or --help boolean: Displays this information. -E or --environment_modules: Include Environment Modules. -e or --easybuild_install_base char: Easybuild software base. Required. -L or --lmod: Include Lmod. -b or --eb_version char: Easybuild Version - Default 3.1.0. -l or --lmod_version char: Lmod Version - Default 7.3.16. -m or --em_version char: Environment Modules Version - Default 3.2.10.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/easybuilders/easybuild/issues/342#issuecomment-325308541, or mute the thread https://github.com/notifications/unsubscribe-auth/AFTFRmrTf6v_joFur9lLuAF1pu8xUusEks5scoqbgaJpZM4N5xBg .

[jerowe]

I'll get it one of these days. I'm back to working more on infrastructure for the foreseeable future, so this will get done! I also got the conda packages for easybuild and deps to build on the conda-forge CI channel, but I didn't submit the PRs correctly. Once I resubmit this script will be able to use the newest packages for deploying.

On Mon, Aug 28, 2017 at 2:28 PM Jillian Rowe [email protected] wrote:

I think I created new issues. ;-)

On Mon, Aug 28, 2017 at 1:38 PM Kenneth Hoste [email protected] wrote:

Hmm... Still the same issue?

$ md5 ~/work/WIP/easybuild_bootstrap_install.sh MD5 (/Users/kehoste/work/WIP/easybuild_bootstrap_install.sh) = 625e330f4ed0f67366c9ab8e58bcbf73 $ ~/work/WIP/easybuild_bootstrap_install.sh -e $PWD easybuild_base is required EasyBuild Bootstrap - Bootstrap method of installing easybuild Usage example: easybuild-bootstrap (-e|--easybuild_install_base) char [(-h|--help) boolean] [(-E|--environment_modules)] [(-L|--lmod)] [(-b|--eb_version) char] [(-l|--lmod_version) char] [(-m|--em_version) char] Options: -h or --help boolean: Displays this information. -E or --environment_modules: Include Environment Modules. -e or --easybuild_install_base char: Easybuild software base. Required. -L or --lmod: Include Lmod. -b or --eb_version char: Easybuild Version - Default 3.1.0. -l or --lmod_version char: Lmod Version - Default 7.3.16. -m or --em_version char: Environment Modules Version - Default 3.2.10.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/easybuilders/easybuild/issues/342#issuecomment-325308541, or mute the thread https://github.com/notifications/unsubscribe-auth/AFTFRmrTf6v_joFur9lLuAF1pu8xUusEks5scoqbgaJpZM4N5xBg .

[Flamefire]

Are you still working on this? I'm not sure if people would actually need this as it is installing anaconda (which is quite heavy...) and there is a bootstrap script which requires just python

If you are: better open a PR so one can comment on code. E.g. if [[ $EASYBUILD_BASE == 0 ]] looks wrong. Also there are mixes of -eq and == and ;then vs \n then. I'd also recommend using shellcheck (e.g. via VSCode plugin) to check for common issues in bash scripts.