vcf2maf
vcf2maf copied to clipboard
mskcc
Docker build fails due to hardcoded Clear Linux version
Hi, when trying to build docker image I got an error that is (I think) caused by lack of backward compatibility between the builder image (clearlinux:latest) and the pinned version Clear Linux that we are trying to install (CLEAR_VERSION=41780). I presume there must have been some change introduced to the clearlinux:latest since you last updated the Dockerfile, which makes it incompatible with the OS version 41780. After I changed to CLEAR_VERSION=42950 the build succeeded.
Here's the error.
$ docker build -t vcf2maf:main .
[+] Building 17.0s (6/13) docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.45kB 0.0s
=> [internal] load metadata for docker.io/library/clearlinux:latest 1.9s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 1.00kB 0.0s
=> [builder 1/4] FROM docker.io/library/clearlinux:latest@sha256:78a9f76be5297fe4455e29169cdd2f12f27e0e093468 11.7s
=> => resolve docker.io/library/clearlinux:latest@sha256:78a9f76be5297fe4455e29169cdd2f12f27e0e093468354933db7 0.0s
=> => sha256:78a9f76be5297fe4455e29169cdd2f12f27e0e093468354933db78f6921f6c50 1.23kB / 1.23kB 0.0s
=> => sha256:ccaf55331ac4d8c815e158493959d73342d631d325badfb789e98d7f84d335b9 1.22kB / 1.22kB 0.0s
=> => sha256:36b6bf15d1c7239e4e1b035b2e702340bd80352a5f0c117ee8a16200fc84f3c4 1.19kB / 1.19kB 0.0s
=> => sha256:d8b446266af417203ddeabfe64ef9ec39bd9602c347380c3e611ddd70af07332 72.25MB / 72.25MB 5.8s
=> => sha256:457f4d5b67f8e66ba8be9f3ca344480f0283e4837e4ce3bd6e1e8c9dfb3cd84a 214B / 214B 0.6s
=> => extracting sha256:d8b446266af417203ddeabfe64ef9ec39bd9602c347380c3e611ddd70af07332 5.7s
=> => extracting sha256:457f4d5b67f8e66ba8be9f3ca344480f0283e4837e4ce3bd6e1e8c9dfb3cd84a 0.0s
=> ERROR [builder 2/4] RUN swupd os-install --no-progress --no-boot-update --no-scripts --version 41780 3.3s
------
> [builder 2/4] RUN swupd os-install --no-progress --no-boot-update --no-scripts --version 41780 --path /install_root --statedir /swupd-state --bundles os-core-update,which:
0.488 Installing OS version 41780
2.163 Error: Mismatching formats detected when diagnosing 41780 (expected: 40; actual: 37)
Latest supported version to diagnose: 42950
3.004
3.004 Installation failed
------
Dockerfile:5
--------------------
4 | ENV CLEAR_VERSION=41780
5 | >>> RUN swupd os-install --no-progress --no-boot-update --no-scripts \
6 | >>> --version ${CLEAR_VERSION} \
7 | >>> --path /install_root \
8 | >>> --statedir /swupd-state \
9 | >>> --bundles os-core-update,which
10 |
--------------------
I can create a PR, but could you first check if the problem is reproducible at your end?
The tests pass.
$ perl tests/vcf2maf.t
1..6
ok 1
ok 2
STATUS: Running VEP and writing to: tests/test_b38.vep.vcf
ok 3
ok 4
STATUS: Running VEP and writing to: tests/test_b38.vep.vcf
ok 5
ok 6
$ perl tests/vcf2vcf.t
1..4
ok 1
ok 2
ok 3
ok 4
$ perl tests/maf2vcf.t
1..4
ok 1
ok 2
ok 3
ok 4
I'm getting the same error:
$ docker build -t vcf2maf:main .
[+] Building 1.7s (6/13) docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.45kB 0.0s
=> [internal] load metadata for docker.io/library/clearlinux:latest 0.4s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> CACHED [builder 1/4] FROM docker.io/library/clearlinux:latest@sha256:649b29eca8296b9b05f18136080165ff42d1d010881fdc68fafa3ee70ac6c3aa 0.0s
=> [internal] load build context 0.1s
=> => transferring context: 12.21MB 0.1s
=> ERROR [builder 2/4] RUN swupd os-install --no-progress --no-boot-update --no-scripts --version 41780 --path /install_root 1.2s
------
> [builder 2/4] RUN swupd os-install --no-progress --no-boot-update --no-scripts --version 41780 --path /install_root --statedir /swupd-state --bundles os-core-update,which:
0.189 Installing OS version 41780
0.287 Error: Mismatching formats detected when diagnosing 41780 (expected: 40; actual: 37)
Latest supported version to diagnose: 43150
1.165
1.165 Installation failed
------
Dockerfile:5
--------------------
4 | ENV CLEAR_VERSION=41780
5 | >>> RUN swupd os-install --no-progress --no-boot-update --no-scripts \
6 | >>> --version ${CLEAR_VERSION} \
7 | >>> --path /install_root \
8 | >>> --statedir /swupd-state \
9 | >>> --bundles os-core-update,which
10 |
--------------------
ERROR: failed to solve: process "/bin/sh -c swupd os-install --no-progress --no-boot-update --no-scripts --version ${CLEAR_VERSION} --path /install_root --statedir /swupd-state --bundles os-core-update,which" did not complete successfully: exit code: 14
I also implemented your fix and the tests passed. However this was after running into perl issues running tests/vcf2maf.t (output has been shortened):
(perl_env2) kevin@kevin-Latitude-5520:~/sw/vcf2maf$ perl tests/vcf2maf.t
1..6
ok 1
ok 2
STATUS: Running VEP and writing to: tests/test_b38.vep.vcf
Can't locate List/MoreUtils.pm in @INC (you may need to install the List::MoreUtils module) (@INC contains: /usr/local/share/ensembl-vep-112.0-0/modules /usr/local/share/ensembl-vep-112.0-0 /usr/local/lib/perl5/5.32/site_perl /usr/local/lib/perl5/site_perl /usr/local/lib/perl5/5.32/vendor_perl /usr/local/lib/perl5/vendor_perl /usr/local/lib/perl5/5.32/core_perl /usr/local/lib/perl5/core_perl .) at /usr/local/share/ensembl-vep-112.0-0/Bio/EnsEMBL/Variation/DBSQL/VariationFeatureAdaptor.pm line 102.
BEGIN failed--compilation aborted at /usr/local/share/ensembl-vep-112.0-0/Bio/EnsEMBL/Variation/DBSQL/VariationFeatureAdaptor.pm line 102.
Compilation failed in require at /usr/local/share/ensembl-vep-112.0-0/modules/Bio/EnsEMBL/VEP/BaseVEP.pm line 59.
BEGIN failed--compilation aborted at /usr/local/share/ensembl-vep-112.0-0/modules/Bio/EnsEMBL/VEP/BaseVEP.pm line 59.
Compilation failed in require at /usr/local/lib/perl5/vendor_perl/base.pm line 100.
...propagated at /usr/local/lib/perl5/vendor_perl/base.pm line 109.
BEGIN failed--compilation aborted at /usr/local/share/ensembl-vep-112.0-0/modules/Bio/EnsEMBL/VEP/BaseRunner.pm line 56.
Compilation failed in require at /usr/local/lib/perl5/vendor_perl/base.pm line 100.
...propagated at /usr/local/lib/perl5/vendor_perl/base.pm line 109.
BEGIN failed--compilation aborted at /usr/local/share/ensembl-vep-112.0-0/modules/Bio/EnsEMBL/VEP/Runner.pm line 71.
Compilation failed in require at /usr/local/bin/vep line 20.
BEGIN failed--compilation aborted at /usr/local/bin/vep line 20.
ERROR: Failed to run the VEP annotator! Command: /usr/local/bin/perl '/usr/local/bin/vep' --species homo_sapiens --assembly GRCh38 --no_progress --no_stats --buffer_size 5000 --sift b --ccds --uniprot --hgvs --symbol --numbers --domains --gene_phenotype --canonical --protein --biotype --uniprot --tsl --variant_class --shift_hgvs 1 --check_existing --total_length --allele_number --no_escape --xref_refseq --failed 1 --vcf --flag_pick_allele --pick_order canonical,tsl,biotype,rank,ccds,length --dir 'tests' --fasta 'tests/Homo_sapiens.GRCh38.dna.chromosome.21.fa' --format vcf --input_file 'tests/test_b38.vcf' --output_file 'tests/test_b38.vep.vcf' --force_overwrite --offline --pubmed --fork 4 --polyphen b --af --af_1kg --af_gnomad --regulatory
.
.
.
# Looks like you failed 4 tests of 6.
I fixed this by adding a conda install step to install perl-list-moreutils to the Dockerfile:
FROM clearlinux:latest AS builder
# Install a minimal versioned OS into /install_root, and bundled tools if any
ENV CLEAR_VERSION=42950
RUN swupd os-install --no-progress --no-boot-update --no-scripts \
--version ${CLEAR_VERSION} \
--path /install_root \
--statedir /swupd-state \
--bundles os-core-update,which
# Download and install conda into /usr/bin
ENV MINICONDA_VERSION=py312_24.4.0-0
RUN curl -sL https://repo.anaconda.com/miniconda/Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh -o /tmp/miniconda.sh && \
bash /tmp/miniconda.sh -bup /usr && \
rm -f /tmp/miniconda.sh && \
conda config --set solver libmamba
# Use mamba to install remaining tools/dependencies into /usr/local
ENV VEP_VERSION=112.0 \
HTSLIB_VERSION=1.20 \
BCFTOOLS_VERSION=1.20 \
SAMTOOLS_VERSION=1.20 \
LIFTOVER_VERSION=447
RUN conda create -y -p /usr/local && \
conda install -y -p /usr/local \
-c conda-forge \
-c bioconda \
-c defaults \
ensembl-vep==${VEP_VERSION} \
htslib==${HTSLIB_VERSION} \
bcftools==${BCFTOOLS_VERSION} \
samtools==${SAMTOOLS_VERSION} \
ucsc-liftover==${LIFTOVER_VERSION}
RUN conda install -y -p /usr/local -c conda-forge -c bioconda perl-list-moreutils
# Deploy the minimal OS and tools into a clean target layer
FROM scratch
LABEL maintainer="Cyriac Kandoth <[email protected]>"
COPY --from=builder /install_root /
COPY --from=builder /usr/local /usr/local
COPY data /opt/data
COPY *.pl /opt/
WORKDIR /opt
I also ran into this issue trying to build the image. I initially thought it was because clear linux is, as of last week, no longer supported (https://community.clearlinux.org/t/all-good-things-come-to-an-end-shutting-down-clear-linux-os/10716), and was initially planning on opening a separate issue for it, but see the base image is already under discussion.
Thanks all. Correct, Clear Linux has been abandoned by Intel. Their work with performance optimizations have been absorbed by other distributions, mostly CachyOS - and I have switched the base OS to this. Please use the new Dockerfile in the main branch. At a later date, I'll push this out as a new release.
Edit: Switched to Debian base OS for the multi-stage build of much smaller size.
