Conda env does not activate at start even if activate=true

[jsmentch]

I'm unable to get my conda env to activate by default with --miniconda activate=True and need to manually enter source activate neuro upon starting the singularity container with singularity shell . I'd like to run the container from a bash/slurm script and it will not allow me to activate the env.

@satra was helping to look into this and this does work on a docker image based on ubuntu

Apparently the line sed -i '$isource activate neuro' $ND_ENTRYPOINT is not executing

Neurodocker command: singularity run neurodocker.simg generate singularity --base centos:7 --pkg-manager yum --user neuro --miniconda create_env=neuro activate=True conda_install="numpy pandas scipy scikit-learn matplotlib seaborn networkx ipython joblib nibabel datalad jupyter numba" pip_install="nibabel ply neurosynth nilearn pliers kmapper dyneusr pyns umap-learn librosa pydra tensorflow pysoundfile resampy" --install emacs vim --run "python -m pip install git+https://github.com/nipype/pydra-ml.git" > Singularity

Build Spec:

# Generated by: Neurodocker version 0.7.0+0.gdc97516.dirty
# Latest release: Neurodocker version 0.7.0
# Timestamp: 2020/05/22 20:04:12 UTC
# 
# Thank you for using Neurodocker. If you discover any issues
# or ways to improve this software, please submit an issue or
# pull request on our GitHub repository:
# 
#     https://github.com/ReproNim/neurodocker

Bootstrap: docker
From: centos:7

%post
su - root

export ND_ENTRYPOINT="/neurodocker/startup.sh"
yum install -y -q \
    bzip2 \
    ca-certificates \
    curl \
    localedef \
    unzip
yum clean packages
rm -rf /var/cache/yum/*
localedef -i en_US -f UTF-8 en_US.UTF-8
chmod 777 /opt && chmod a+s /opt
mkdir -p /neurodocker
if [ ! -f "$ND_ENTRYPOINT" ]; then
  echo '#!/usr/bin/env bash' >> "$ND_ENTRYPOINT"
  echo 'set -e' >> "$ND_ENTRYPOINT"
  echo 'export USER="${USER:=`whoami`}"' >> "$ND_ENTRYPOINT"
  echo 'if [ -n "$1" ]; then "$@"; else /usr/bin/env bash; fi' >> "$ND_ENTRYPOINT";
fi
chmod -R 777 /neurodocker && chmod a+s /neurodocker

test "$(getent passwd neuro)" || useradd --no-user-group --create-home --shell /bin/bash neuro
su - neuro

export PATH="/opt/miniconda-latest/bin:$PATH"
echo "Downloading Miniconda installer ..."
conda_installer="/tmp/miniconda.sh"
curl -fsSL --retry 5 -o "$conda_installer" https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash "$conda_installer" -b -p /opt/miniconda-latest
rm -f "$conda_installer"
conda update -yq -nbase conda
conda config --system --prepend channels conda-forge
conda config --system --set auto_update_conda false
conda config --system --set show_channel_urls true
sync && conda clean -y --all && sync
conda create -y -q --name neuro
conda install -y -q --name neuro \
    "numpy" \
    "pandas" \
    "scipy" \
    "scikit-learn" \
    "matplotlib" \
    "seaborn" \
    "networkx" \
    "ipython" \
    "joblib" \
    "nibabel" \
    "datalad" \
    "jupyter" \
    "numba"
sync && conda clean -y --all && sync
bash -c "source activate neuro
  pip install --no-cache-dir  \
      "nibabel" \
      "ply" \
      "neurosynth" \
      "nilearn" \
      "pliers" \
      "kmapper" \
      "dyneusr" \
      "pyns" \
      "umap-learn" \
      "librosa" \
      "pydra" \
      "tensorflow" \
      "pysoundfile" \
      "resampy""
rm -rf ~/.cache/pip/*
sync
sed -i '$isource activate neuro' $ND_ENTRYPOINT


yum install -y None \
    emacs \
    vim
yum clean packages
rm -rf /var/cache/yum/*

python -m pip install git+https://github.com/nipype/pydra-ml.git

echo '{
\n  "pkg_manager": "yum",
\n  "instructions": [
\n    [
\n      "base",
\n      "centos:7"
\n    ],
\n    [
\n      "user",
\n      "root"
\n    ],
\n    [
\n      "_header",
\n      {
\n        "version": "generic",
\n        "method": "custom"
\n      }
\n    ],
\n    [
\n      "user",
\n      "neuro"
\n    ],
\n    [
\n      "miniconda",
\n      {
\n        "create_env": "neuro",
\n        "activate": true,
\n        "conda_install": [
\n          "numpy",
\n          "pandas",
\n          "scipy",
\n          "scikit-learn",
\n          "matplotlib",
\n          "seaborn",
\n          "networkx",
\n          "ipython",
\n          "joblib",
\n          "nibabel",
\n          "datalad",
\n          "jupyter",
\n          "numba"
\n        ],
\n        "pip_install": [
\n          "nibabel",
\n          "ply",
\n          "neurosynth",
\n          "nilearn",
\n          "pliers",
\n          "kmapper",
\n          "dyneusr",
\n          "pyns",
\n          "umap-learn",
\n          "librosa",
\n          "pydra",
\n          "tensorflow",
\n          "pysoundfile",
\n          "resampy"
\n        ]
\n      }
\n    ],
\n    [
\n      "install",
\n      [
\n        "emacs",
\n        "vim"
\n      ]
\n    ],
\n    [
\n      "run",
\n      "python -m pip install git+https://github.com/nipype/pydra-ml.git"
\n    ]
\n  ]
\n}' > /neurodocker/neurodocker_specs.json

%environment
export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"
export ND_ENTRYPOINT="/neurodocker/startup.sh"
export CONDA_DIR="/opt/miniconda-latest"
export PATH="/opt/miniconda-latest/bin:$PATH"

%runscript
/neurodocker/startup.sh "$@"

[stebo85]

Dear @jsmentch,

Could your problem be similar to https://github.com/ReproNim/neurodocker/issues/346 and issue https://github.com/ReproNim/neurodocker/issues/242 be an explanation for the behavior you are seeing ?

It seems that the startup script activating the environemnts gets called on run but not exec.

[satra]

@stebo85 - in this case it seems like the sed instruction to add activate to the startup script is not working. i downloaded the singularity image that jsmentch posted and i can see the instruction in the file, but not in the startup script. strangely if i run it in the singularity container itself (in tmpfs write mode) it adds it properly to the startup script.

[kaczmarj]

thanks for the report @jsmentch

i can reproduce the error that neuro is not activated. i noticed that when i ran conda env list inside the container, my local machine's conda environments appeared. so i reran the singularity container with the --contain flag, and the issue is fixed. that flag empties the $HOME directory, which is where my miniconda installation exists.

jakub@dash:/tmp/foobar$ singularity run --contain jk.sif 
Singularity> which python
/opt/miniconda-latest/envs/neuro/bin/python

jakub@dash:/tmp/foobar$ singularity run jk.sif 
Singularity> which python
/home/jakub/miniconda3/bin/python

three points about the singularity definition:

1. i ran into an error building the singularity image on the step that install pydra-ml. git is not installed in the image, so that pip install command failed. you can fix that by installing pydra-ml from the url https://github.com/nipype/pydra-ml/tarball/master.
2. pydra-ml depends on shap, which has some pieces that need to be compiled. the compilation fails because gcc is not installed. instead, it's quicker to install shap from conda because conda ships pre-compiled packages.
3. if you only intend to use the neuro environment in this container, you might as well install things into the base environment (then you will only have one environment and nothing to activate).

here is an updated neurodocker generate that reflects the changes from the three points above.

singularity run neurodocker.simg generate singularity \
  --base centos:7 --pkg-manager yum \
  --user neuro \
  --miniconda use_env=base \
      conda_install="numpy pandas scipy scikit-learn matplotlib seaborn networkx ipython joblib nibabel datalad jupyter numba shap" \
      pip_install="nibabel ply neurosynth nilearn pliers kmapper dyneusr pyns umap-learn librosa pydra tensorflow pysoundfile resampy https://github.com/nipype/pydra-ml/tarball/master" \
  --user root \
  --install emacs vim > Singularity 

[jsmentch]

Installing into the base environment makes more sense for me and for now this is all working - thank you for your help and these tips!

[djarecka]

anyone is working on this?

[kaczmarj]

@djarecka - i don't think so. i do not have time right now to work on this.

[djarecka]

@kaczmarj - thanks, I'll give a try

[djarecka]

I think we don't have activate=true anymore, so this is probably not relevant