Add feature to upload Docker images to AWS [WIP]

[davidjurado]

New command for MLCube has been added:

mlcube upload

The idea of this command is to allow users to upload the docker image specified inside the mlcube.yaml file to different registries, at the moment, this PR contains the logic to upload images to ECR (service from AWS).

This command expects to have the following AWS credentials as environment variables:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_REGION

Another way to provide the credentials is providing an aws_credentials.json file containing the credentials with the following structure:

{
        "access_key_id": "XXXXXXXXXX",
        "secret_access_key": "XXXXXXXXXX",
        "region": "XXXXXXXXXX"
}

Note: this file must be located in the same folder of the mlcube.yaml

How to test this feature?

Install the latest version of the Docker runner:

virtualenv -p python3 env && source ./env/bin/activate

git clone https://github.com/mlcommons/mlcube && cd ./mlcube

git fetch origin pull/244/head:feature/upload_image_to_aws

git checkout feature/upload_image_to_aws

pip install semver spython && pip install ./mlcube

pip install --no-deps --force-reinstall ./runners/mlcube_docker

Then configure the AWS credentials, after that run:

mlcube upload

[github-actions[bot]]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[dfeddema]

You can do this with an open source tool called skopeo. e.g. aws ecr create-repository --repository-name '{{ NEW REPO NAME }} ' && skopeo copy --format=v2s2 containers-storage:localhost/{{ my-podman-created-image}}:{{ image-tag}} docker://{{ AWS ACCOUNT }}.dkr.ecr.{{ AWS REGION }}.amazonaws.com/{{ NEW REPO NAME }} --dest-creds AWS:$(aws ecr get-login-password --output text)