Improve docker build scripts

[vfdev-5]

🚀 Feature

Currently, we can build and publish dockers using Circle CI and it works more or less well, however, PR's modifying Dockerfile can not build docker images to ensure PR correctness (related to the way how the PR is checkout on Circle CI).

The idea is

• refactor all build_all.sh scripts (docker/main/build_all.sh, docker/hvd/build_all.sh and docker/msdp/build_all.sh) into a single shell script that could build a single docker image.
• use github actions to build only on PR a single docker image and introduce a single yml file for that (to see what can be done).
  • publishing will use existing scripts
• add a test inside docker folder as a python script to check docker image versions for : torch, ignite ; can import cv2 ; maybe something else.

To Do list:

• [x] refactor build_all.sh into build.sh
• [x] run docker image build for PR on GA
• [ ] run tests on built image. Tests
  • For all images
    • can import torch and its version == required one
    • can import ignite and its version == required one
  • for all -vision images
    • can import opencv without driver issue
  • for all horovod images
    • can import horovod and its version == required one
  • for all msdp images
    • can import deepspeed and its version == required one

cc @fco-dv

[sparkingdark]

I am not sure about the GitHub actions, that build the docker image from the pr, but I think the first point is an idea to merge all shell file to one main shell script.

what is the difference between the folders main , hvd, and msdp

[vfdev-5]

what is the difference between the folders main , hvd, and msdp

@sparkingdark checkout https://github.com/pytorch/ignite/tree/master/docker and installation section. It is about having different frameworks in addition to pytorch and ignite.

[sparkingdark]

Okay I will check that.

[vfdev-5]

@trsvchn would you like to tackle this issue ?

[trsvchn]

@vfdev-5 sure!

[vfdev-5]

I'll start on this point

• refactor all build_all.sh scripts (docker/main/build_all.sh, docker/hvd/build_all.sh and docker/msdp/build_all.sh) into a single shell script that could build a single docker image.

If you could help with:

• create gtihub actions
• add a test inside docker folder as a python script to check docker image versions for : torch, ignite ; can import cv2 ; maybe something else.

would be great !

[trsvchn]

@vfdev-5 Yes, I am going to start from this:

• create gtihub actions
• add a test inside docker folder as a python script to check docker image versions for : torch, ignite ; can import cv2 ; maybe something else.

[vfdev-5]

One another thing I was also thinking about is about building docker images on "release" event. Currently, everything happens in parallel:

• binary distributions to pypi and conda
• docker image build

As "binary distributions to pypi and conda" faster than "docker image build", layer responsible for fetching the lastest ignite can get the released version. Maybe, we should force to wait "docker image build"...

[trsvchn]

@vfdev-5 I think it possible to trigger workflow manually, so we can trigger "docker image build" after "binary distributions to pypi and conda"

[vfdev-5]

I'm not that fan of manual trigger, I was thinking about naive "check last version and wait 1 minute"...

[trsvchn]

@vfdev-5 Oh, no "manually" means using a kind workaround with manual events

https://docs.github.com/en/actions/reference/events-that-trigger-workflows#manual-events

I've found this solution

https://github.community/t/trigger-an-action-upon-completion-of-another-action/17642/2

[vfdev-5]

That's could be great !

[Asrst]

@vfdev-5,

Hi, Can I work on task 3 to close this issue (add a test inside docker folder as a python script to check docker image versions for : torch, ignite ; can import cv2) ?

[vfdev-5]

@Asrst thanks for asking. Actually, we already did that here: https://github.com/pytorch/ignite/blob/master/docker/test_image.py I'll close this issue as done.

if you would like to contribute, please pick another "help wanted" issue: https://github.com/pytorch/ignite/labels/help wanted