Add script to automate updating conda environments in Docker
Adds a script create_spec_file.py that can be used to regenerate reproducible conda environment spec files for use with NISAR ADT Docker images.
Background
- NISAR ADT Docker images are built using a multi-stage Docker build that separates the installation of ISCE3's build-time dependencies from its runtime dependencies.
- ISCE3 itself is installed in the "base" conda environment. The SoilMoisture software and its dependencies are installed in a separate conda environment.
- ISCE3's build-time and runtime dependencies are listed in separate
requirements.txtfiles here and here. However, these requirements files aren't directly used to instantiate the conda environment inside the Docker image. Instead, we create "spec files" that explicitly list the exact packages inside the conda environment in order to ensure that builds are reproducible (and to speed up installation). These spec files are stored separately in the repository (here and here). - We similarly list the direct dependencies of the SoilMoisture software in a conda environment YAML file here and use a separate spec file (found here) to instantiate the SoilMoisture environment in Docker images.
- Recently, in #58, we began installing CUDA dependencies using conda instead of yum. However, the CUDA dependencies aren't listed in the
requirements.txtfiles currently, since they're installed from Nvidia's conda channel instead of conda-forge.
Updates
In order to facilitate fetching packages from multiple conda channels (conda-forge and nvidia), the requirements.txt files were removed and replaced with environment.yml files. The new environment.yml files include the CUDA dependencies and list the nvidia/label/cuda-11.8.0 channel in addition to the conda-forge channel.
-
cuda-cudartandlibcufftwere added to the list of runtime dependencies. -
cuda-cudart-dev,cuda-nvcc,libcufft-dev, andrpm-toolswere added to the list of build-time dependencies.
A new executable Python script was added to automate creation of spec files from these environment.yml files. The script has three different modes: runtime creates a spec file with ISCE3's runtime dependencies, dev creates a spec file with build-time dependencies, and soil-moisture creates a spec file with the SoilMoisture dependencies. Updating all of the spec files requires running the script three times, e.g.
$ ./tools/create_spec_file.py runtime
$ ./tools/create_spec_file.py dev
$ ./tools/create_spec_file.py soil-moisture