Proposal of a Singularity build environment

[maxaehle]

Summary

I'd like to contribute the content of this repository:
https://github.com/maxaehle/gate-singularity

It contains a definition file to build a Singularity container, and the download/build commands of the Dockerfiles in Gate/source/docker as separate scripts. It is meant to assist development on a system without root access. (Opposed to this, I think the Dockerfiles are meant to produce a running version once, i.e. are made for users not developers.)

Question: If you consider this useful, please help me contributing it. Should I make a pull request? Which repository and directory do you consider most appropriate?

Further details

Why Singularity instead of Docker?

I'd like to compile GATE and its dependencies on a node of a cluster. Docker has not been designed for this kind of environment and is therefore not available on the nodes accessible to me. Singularity was designed for use on clusters. Building the Singularity image requires root privileges, but running a read-only container doesn't.

But I can also provide an additional Dockerfile counterpart to the Singularity definition file, if you like.

Why are the compile commands not part of the definition file, but separate scripts?

• The Singularity image must be built on a local machine as it requires sudo. As I want to compile GATE(+dependencies) on the remote system, compilation just cannot happen at this stage.
• Additionally, it makes recompilation easier: Just run again (some of) the scripts, possibly after editing them.

Why are the download commands not part of the definition file, but a separate script?

• If the sources were downloaded while building the image, they would become part of the isolated file system of the container. Now they are downloaded while the container is already running, they can be stored on the host file system by starting the container with --bind software:/software. This allows to run the container read-only, and makes editing, version control etc. easier.
• Furthermore, it prevents having to download the sources to the local machine (with a slower internet connection than the cluster).

Why have the CMake/make calls been split into many individual script?

So you can recompile only the subproject that you're working on.

Preferred locations

The definition file and download/build scripts (see the file hierarchy here) could go into a new directory Gate/source/singularity. However I'd prefer to have them outside the Gate repository, because the Gate repository is cloned into a subdirectory software/gate and you would then have two copies of it. So a directory in the GateContrib repository would fit better in my opinion.

[BishopWolf]

Hey, I am one of the gate docker app creators.

I don't get why you think docker has to be run as root and that it can't be run in clusters? That's my daily work!

To run gate docker app in a cluster you have to get your user inside the docker group, the docker itself has to be running as service. And the cluster part: you can share folders among docker apps for writing results in a single folder for instance. Of course you need to take care of not writing the same file with two instances.

Yes, gate docker app is built once (at every release) and is explicitly intended for users.

[maxaehle]

@BishopWolf thanks for your message! Actually, both the compute nodes accessible to me have singularity installed but not docker, and I presumed for this reason, although I see now that there are rootless variants of docker.

I can extend the proposal with a Dockerfile analogous to the Singularity definition file, should people find this useful. The Docker-Singularity difference is only a part of the proposal. Another difference is the intention and where the compilation of GATE(+dependencies) takes place (while creating the image vs. in a running container):

• Gate/source/docker is built once (at every release) and is explicitly intended for users. The compilation takes place during docker build, so the compiled files are part of the image. I agree that for a user of a program, this is the best way to do it. By supplying something like -v hostpath:containerpath to docker run, a user can share the output produced by Gate at run-time, as you write.
• I created the above setup for easy (re-)compilation of the code. singularity build will just install the necessary build tools, and the compilation takes place by invoking a shell script in a container started with singularity run. By supplying something like --bind hostpath:containerpath to docker run, the developer can make the compilation output persistent (to speed up recompilation). I think this could be more suitable for development. It gives you the option to interactively enter any build commands in the command line, instead of editing the Dockerfile and building the whole image again. For me this was quite useful to perform tests similar to #480.