Reusing environment variables defined by other modules

[AreWeDreaming]

I have encountered the following situation and I could not find an immediate solution in the documentation. Assume I have a module A:

setenv("NETCDF_DIR","<path to NetCDF>")

and a module B for a legacy application that depends on A :

depends_on("A")
setenv("NETCDF_BUILD_ARG","-L" .. os.getenv("NETCDF_DIR") .. "/lib")

Now I can module load B just fine, but when I module purge $NETCDF_BUILD_ARG remains defined because $NETCDF_DIR becomes undefined before $NETCDF_BUILD_ARG can be unset. I understand that this problem could be solved by modifying module A such that it reads

setenv("NETCDF_DIR","<path to NetCDF>")
prepend_path("MODULEPATH", "<path containing module B>")

and isolate B to a separate folder and remove the depends_on statement, but unfortunately this solution would be difficult to implement in our current circumstances.

So my question is, if structuring modules hierarchically is not an option, is there a recommended way of how modules can use environment variables that have been defined by other modules? Thank you for your help in advance!

[rtmclay]

As you have figured out, os.getenv("FOO") returns nil if FOO is not in the environment. Also that Lua errors out if you try to concatenate a nil and a string. In this case you can solve this by doing the following:

depends_on("A")
local netcdf_dir = os.getenv("NETCDF_DIR") or ""
setenv("NETCDF_BUILD_ARG","-L" .. netcdf_dir .. "/lib")

You can do this when unloading a module the value in setenv(name,value) does not matter because Lmod causes name to be unset.

In general, you might have to have a common function implemented in SitePackage.lua or /etc/lmod/lmod_config.lua that both A and B call to define a env. var like NETCDF_DIR.

[rtmclay]

I have found that it is better to use the introspection functions like myModuleFullName() and myFileName() to figure out where the modulefile is in the module tree. With that information, it might be possible to where the application/library like NetCDF is stored in the the system directory tree.

You might find this discussion helpful: https://lmod.readthedocs.io/en/latest/100_modulefile_examples.html#generic-modules-with-the-hierarchy

[rtmclay]

O.K. to close this issue?

[AreWeDreaming]

First, I would thank you for your advice. Your initial solution is workable, and I will take a look at the introspection. Thank you for that and sorry for not getting back to you sooner.

There was one other thing similar to this and that is the usage of pushenv.

We have a module for our compilers called gcc and then an extension module mpi that adds MPI on top of that and is dependent (depends_on) on gcc. The gcc module defines (among other things) CC=gcc, FC=gfortran and the like, and the MPI module then uses pushenv to overwrite these to CC=mpicc and FC=mpifort.

When env is purged it unloads its dependency gcc first which unsets CC and FC as expected. However, once we get to the pushenv commands in the mpi module it restores the CC and FC variables to the values set by the gcc module. What is an elegant solution to solve this?

[rtmclay]

The best solution is to use the software hierarchy and you avoid this problem entirely.

If you are not using the hierarchy then you might try creating a module called say "gcc_helper" which has everything that gcc sets except for setting pushenv("CC","gcc"). I have ignored setting FC here.

-- all path info for gcc
prepend_path("PATH","/path/to/gcc/bin")
...

Then gcc is just:

depends_on("gcc_helper")
prepend_path("CC","gcc")

And mpi is

depends_on("gcc_helper")
prepend_path("CC","mpicc")

I have tested this alittle and it mostly works but doing:

$ module load mpi gcc

will lead to CC pointing to gcc and not mpicc.

No promises that this will work in all cases. If you can't make setting CC and FC reliable, I would not set them.

[rtmclay]

Now is it O.K. to close?

[AreWeDreaming]

Yes, thank you.