isce3 icon indicating copy to clipboard operation
isce3 copied to clipboard

Add script to automate updating conda environments in Docker

Open gmgunter opened this issue 4 months ago • 0 comments

Adds a script create_spec_file.py that can be used to regenerate reproducible conda environment spec files for use with NISAR ADT Docker images.

Background

  • NISAR ADT Docker images are built using a multi-stage Docker build that separates the installation of ISCE3's build-time dependencies from its runtime dependencies.
  • ISCE3 itself is installed in the "base" conda environment. The SoilMoisture software and its dependencies are installed in a separate conda environment.
  • ISCE3's build-time and runtime dependencies are listed in separate requirements.txt files here and here. However, these requirements files aren't directly used to instantiate the conda environment inside the Docker image. Instead, we create "spec files" that explicitly list the exact packages inside the conda environment in order to ensure that builds are reproducible (and to speed up installation). These spec files are stored separately in the repository (here and here).
  • We similarly list the direct dependencies of the SoilMoisture software in a conda environment YAML file here and use a separate spec file (found here) to instantiate the SoilMoisture environment in Docker images.
  • Recently, in #58, we began installing CUDA dependencies using conda instead of yum. However, the CUDA dependencies aren't listed in the requirements.txt files currently, since they're installed from Nvidia's conda channel instead of conda-forge.

Updates

In order to facilitate fetching packages from multiple conda channels (conda-forge and nvidia), the requirements.txt files were removed and replaced with environment.yml files. The new environment.yml files include the CUDA dependencies and list the nvidia/label/cuda-11.8.0 channel in addition to the conda-forge channel.

  • cuda-cudart and libcufft were added to the list of runtime dependencies.
  • cuda-cudart-dev, cuda-nvcc, libcufft-dev, and rpm-tools were added to the list of build-time dependencies.

A new executable Python script was added to automate creation of spec files from these environment.yml files. The script has three different modes: runtime creates a spec file with ISCE3's runtime dependencies, dev creates a spec file with build-time dependencies, and soil-moisture creates a spec file with the SoilMoisture dependencies. Updating all of the spec files requires running the script three times, e.g.

$ ./tools/create_spec_file.py runtime
$ ./tools/create_spec_file.py dev
$ ./tools/create_spec_file.py soil-moisture

gmgunter avatar Sep 24 '25 06:09 gmgunter