Feature Request - embed HPCCM recipe files in definition/Dockerfile

Open laytonjbgmail opened this issue 5 years ago • 0 comments

This ia feature request.

Singularity stores its definition file in the resulting image. This can be extracted using "singularty inspect -d". This allows you to see how the image was created. You can then edit this definition and create a new image.

I don't like editing Singularity definition files or Dockerfiles directly (they can be too long and wildly complicationed). I like editing HPCCM recipe files. However, they are not stored with the images.

I would like to see a way to store HPCCM recipes in the Singularity of Docker images. (similiar to how Singularity stores it's definition file).

For example, Singularity and Docker both have the capability of storing metadata in a section like "%label". It would be useful to store the HPCCM recipe in the %label section of the images.

Here is a simple recipe:

Stage0 += baseimage(image='nvidia/cuda:10.1-base-ubuntu18.04')

Stage0 += shell(commands=['apt-get update'])
Stage0 += shell(commands=['apt-get install -y octave'])

Stage0 += pgi(eula=True, mpi=True)

Using HPCCM to "process" this definition file,

$ hpccm --recipe test2.py --format singularity > test2.def

Produces a definition file that looks like,

BootStrap: docker From: nvidia/cuda:10.1-base-ubuntu18.04 %post . /.singularity.d/env/10-docker*.sh

%post cd / apt-get update

%post cd / apt-get install -y octave

PGI compiler version 19.10

%post apt-get update -y DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends
g++
gcc
libnuma1
openssh-client
perl
wget rm -rf /var/lib/apt/lists/* %post cd / mkdir -p /var/tmp && wget -q -nc --no-check-certificate -O /var/tmp/pgi-community-linux-x64-latest.tar.gz --referer https://www.pgroup.com/products/community.htm?ut m_source=hpccm&utm_medium=wgt&utm_campaign=CE&nvid=nv-int-14-39155 -P /var/tmp https://www.pgroup.com/support/downloader.php?file=pgi-community-linux-x64 mkdir -p /var/tmp/pgi && tar -x -f /var/tmp/pgi-community-linux-x64-latest.tar.gz -C /var/tmp/pgi -z cd /var/tmp/pgi && PGI_ACCEPT_EULA=accept PGI_INSTALL_DIR=/opt/pgi PGI_INSTALL_MPI=true PGI_INSTALL_NVIDIA=true PGI_MPI_GPU_SUPPORT=true PGI_SILENT=true ./install echo "variable LIBRARY_PATH is environment(LIBRARY_PATH);" >> /opt/pgi/linux86-64/19.10/bin/siterc echo "variable library_path is default($if($LIBRARY_PATH,$foreach(ll,$replace($LIBRARY_PATH,":",), -L$ll)));" >> /opt/pgi/linux86-64/19.10/bin/siterc echo "append LDLIBARGS=$library_path;" >> /opt/pgi/linux86-64/19.10/bin/siterc ln -sf /usr/lib/x86_64-linux-gnu/libnuma.so.1 /opt/pgi/linux86-64/19.10/lib/libnuma.so ln -sf /usr/lib/x86_64-linux-gnu/libnuma.so.1 /opt/pgi/linux86-64/19.10/lib/libnuma.so.1 rm -rf /var/tmp/pgi-community-linux-x64-latest.tar.gz /var/tmp/pgi %environment export LD_LIBRARY_PATH=/opt/pgi/linux86-64/19.10/mpi/openmpi-3.1.3/lib:/opt/pgi/linux86-64/19.10/lib:$LD_LIBRARY_PATH export PATH=/opt/pgi/linux86-64/19.10/mpi/openmpi-3.1.3/bin:/opt/pgi/linux86-64/19.10/bin:$PATH %post export LD_LIBRARY_PATH=/opt/pgi/linux86-64/19.10/mpi/openmpi-3.1.3/lib:/opt/pgi/linux86-64/19.10/lib:$LD_LIBRARY_PATH export PATH=/opt/pgi/linux86-64/19.10/mpi/openmpi-3.1.3/bin:/opt/pgi/linux86-64/19.10/bin:$PATH

I would like that definition file to look like the following.

BootStrap: docker From: nvidia/cuda:10.1-base-ubuntu18.04 %post . /.singularity.d/env/10-docker*.sh

%post cd / apt-get update

%post cd / apt-get install -y octave

PGI compiler version 19.10

%label Stage0 += baseimage(image='nvidia/cuda:10.1-base-ubuntu18.04') Stage0 += shell(commands=['apt-get update']) Stage0 += shell(commands=['apt-get install -y octave']) Stage0 += pgi(eula=True, mpi=True)

Notice how the HPCCM recipe is stored in the %label section. Now I can extract it from an image created with the definition file and edit that recipe.

Thanks!

May 04 '20 21:05 laytonjbgmail