aws
[ECR] [Replication]: trigger replication when regions are updated
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Tell us about your request What do you want us to build? A trigger that starts replication for newly added regions for "Cross-Region replication".
Which service(s) is this request for? ECR
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? I need that replications start automatically to a region that was just added to the Cross-Region Replication of a registry. Currently, I believe the API is expecting the Cross-Region Replication to be activated (and regions set) prior to the creation and pushing of an image. However, if a new region is added to the same registry, the replication will only occur (for a single repository) if a new image/tag is pushed to a repository. If this cannot be solved, it will impact the deployment of services to new regions.
Are you currently working around this issue?
Original:
~~Currently, the best (easiest) workaround I found is to delete the image (tag) and immediately push it back into the repository. This won't require any actual upload to happen (as the layers will still exist) and will trigger the replication of only the image/tag pushed.~~
Edit 1:
I found a much better way of triggering the replication (it only works for single image tags):
Using the API or SDK of your preference perform a batch-get-image to thoroughly describe all image/tags you want to trigger replication. For each image tag, take the imageManifest from the batch-get-image response and use put-image to update the image tag using the same manifest. This shall trigger the replication as if a new image has just been pushed.
Edit 2:
Actually, the previous approach was pretty hacky as the imageManifest needs to be changed (it's a string) somehow - adding or removing a linebreak was enough to trigger everything. The less hacky approach is to delete the image tag before attempting to update it. I'm pretty sure this is still hacky and might bite me back in the future - that's why this request is important.
Additional context It would be nice to have this feature even before the ability to select repositories or images by tag/regex.
It would be similarly useful to also trigger when new destination accounts are added to the configuration.
Here is an extract of the scripts I end up doing
--- to copy ----
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $ECR_REGISTRY
IMAGE_TAGS=$(aws ecr list-images --repository-name $ECR_REPO --query 'imageIds[].imageTag' --output text)
for image_tag in $IMAGE_TAGS
do
MANIFEST=$(aws ecr batch-get-image --repository-name $ECR_REPO --image-id imageTag=$image_tag --query 'images[].imageManifest' --output text)
new_tag=$image_tag".temp"
echo "copying image $ECR_REPO:$image_tag"
aws ecr put-image --repository-name $ECR_REPO --image-tag $new_tag --image-manifest "$MANIFEST" >/dev/null 2>&1
aws ecr batch-delete-image --repository-name $ECR_REPO --image-ids imageTag=$image_tag >/dev/null 2>&1
aws ecr put-image --repository-name $ECR_REPO --image-tag $image_tag --image-manifest "$MANIFEST" >/dev/null 2>&1
aws ecr batch-delete-image --repository-name $ECR_REPO --image-ids imageTag=$new_tag >/dev/null 2>&1
done
Unfortunately, the tag deletion are not propagated, so I add to delete the ".temp" tag on the destination repo
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $ECR_REGISTRY
IMAGE_TAGS=$(aws ecr list-images --repository-name $ECR_REPO --query 'imageIds[].imageTag' --output text)
for image_tag in $IMAGE_TAGS
do
if [[ $image_tag == *.temp ]]
then
echo "Deleting image $ECR_REPO:$image_tag"
MANIFEST=$(aws ecr batch-get-image --repository-name $ECR_REPO --image-id imageTag=$image_tag --query 'images[].imageManifest' --output text)
aws ecr batch-delete-image --repository-name $ECR_REPO --image-ids imageTag=$image_tag >/dev/null 2>&1
fi
done
Based on previous comment, I writed a python code that do the same thing, but for all repositories on your account.
#!/usr/bin/env python3
import boto3
session = boto3.Session(profile_name='YOUR_AWS_PROFILE_NAME_HERE',region_name='YOUR_SOURCE_REGION_HERE')
ecr = session.client('ecr')
registryId="YOUR_AWS_ACCOUNT_ID_HERE"
repositories_list = ecr.describe_repositories(
registryId=registryId,
maxResults=1000
)
for repository in repositories_list['repositories']:
repositoryName=repository['repositoryName']
imagesList = ecr.list_images(
registryId=registryId,
repositoryName=repositoryName,
maxResults=1000,
filter={
'tagStatus': 'TAGGED'
}
)
for image in imagesList['imageIds']:
imageTag=image['imageTag']
imageData = ecr.batch_get_image(
registryId=registryId,
repositoryName=repositoryName,
imageIds=[
{
'imageTag': imageTag
},
],
)
print("reuploading: ",repositoryName,":",imageTag, end=" ")
for image in imageData['images']:
tempTag=imageTag+"-temp"
imageManifest=image['imageManifest']
ecr.put_image(
registryId=registryId,
repositoryName=repositoryName,
imageManifest=imageManifest,
imageTag=tempTag
)
ecr.batch_delete_image(
registryId=registryId,
repositoryName=repositoryName,
imageIds=[
{
'imageTag': imageTag
},
]
)
ecr.put_image(
registryId=registryId,
repositoryName=repositoryName,
imageManifest=imageManifest,
imageTag=imageTag
)
ecr.batch_delete_image(
registryId=registryId,
repositoryName=repositoryName,
imageIds=[
{
'imageTag': tempTag
},
]
)
print("OK")
and for delete the temp images replicated on destination region
#!/usr/bin/env python3
import boto3
session = boto3.Session(profile_name='YOUR_AWS_PROFILE_NAME_HERE',region_name='YOUR_DESTINATION_REGION_HERE')
ecr = session.client('ecr')
registryId="YOUR_AWS_ACCOUNT_ID_HERE"
repositories_list = ecr.describe_repositories(
registryId=registryId,
maxResults=1000
)
for repository in repositories_list['repositories']:
repositoryName=repository['repositoryName']
imagesList = ecr.list_images(
registryId=registryId,
repositoryName=repositoryName,
maxResults=1000,
filter={
'tagStatus': 'TAGGED'
}
)
for image in imagesList['imageIds']:
imageTag=image['imageTag']
imageData = ecr.batch_get_image(
registryId=registryId,
repositoryName=repositoryName,
imageIds=[
{
'imageTag': imageTag
},
],
)
if imageTag.endswith('-temp'):
print("deleting: {0}:{1}".format(repositoryName,imageTag), end=" ... ")
ecr.batch_delete_image(
registryId=registryId,
repositoryName=repositoryName,
imageIds=[
{
'imageTag': imageTag
},
]
)
print("OK")
This works if Repository Type - "image" but what about the repositories where Repository Type ---"Helm Chart"
Will this also work for Cross Account as well, where other region we are replication is not the Primary region.
Is it needed to enable CAR in other accounts? but that will be unwanted configuration to have CAR also enabled in the regions we need.
Would be nice to have the same functionality as in S3 where we can launch a batch operation when a replication rule is added.
There is also a proposal for ECR to ECR pull-through cache, which could replace full replication. I think it would also address the issue described in this ticket - already existing images in region A would be cached in region B on demand. Also that way you would not replicate images that are maybe no longer needed (saving $$$ on transfer and storage in the new region). If you think it would help - vote: https://github.com/aws/containers-roadmap/issues/2208
