Cached images do not update.

[davidkarlsen]

Describe the bug Images to not re-pull tags that are updated. Additionally - if one deletes a a whole registry path that was cached, one will get a not found when later pulling (i.e. ACR not re-pull from upstream)

To Reproduce Steps to reproduce the behavior:

1. Have a rule like
2. observe images not being updated

ghcr.io/davidkarlsen/utils-image main 3a641dbbef6f 22 hours ago 599MB MYREG.azurecr.io/ghcr.io/davidkarlsen/utils-image main 9c88223a0707 11 days ago 575MB

Expected behavior Should update tags. Should be able to re-pull images that were previously cached but have been deleted.

MYREG.azurecr.io/ghcr.io/davidkarlsen/flyway-operator-catalog:alpha not found: manifest unknown: manifest tagged by "alpha" is not found but docker pull ghcr.io/davidkarlsen/flyway-operator-catalog:alpha will work just fine

Screenshots If applicable, add screenshots to help explain your problem.

Any relevant environment information

• OS: [e.g. Ubuntu, Windows] mac
• Azure CLI/PowerShell/SDK version N/A
• Docker version Docker version 26.1.4, build 5650f9b
• Datetime (UTC) when the issue occurred 29.06.2024 13:22
• Registry and image names ghcr.io

Additional context Add any other context about the problem here.

If any information is a concern to post here, you can create a support ticket or send an email to [email protected].

[luisdlp]

Thanks for reporting the bug. @davidkarlsen can you try to delete and recreate the cache rule? Are you still able to repro? We recently deployed in our cache backend that could have caused this issue. If this still repros, please open a support ticket or send us email with your registry name so that we can investigate further.

[davidkarlsen]

@luisdlp for that I have to drop the existing repos:

"details":[{"code":"TargetRepositoryPrefixMatchesExistingRegistries","message":"Conflict: There are one or more target repositories in the registry which match the target repository prefix 'ghcr.io/*'. Cache rules cannot be created for existing repositories in registry. For more information, please visit https://aka.ms/acr/cache"}]}]}]}}

I'll try that and see.

[davidkarlsen]

@luisdlp

this works:

1. deleting the cache rule
2. and all existing repos created by it
3. and re-creating cache-rule

makes it work again.

But surely a fix is needed in ACR to avoid that?

[luisdlp]

@davidkarlsen We had a regression on a recent deployment which we deployed a fix for last week. Unfortunately, you might need to re-create other rules if you see the same behavior. We apologize for this inconvenience.

[davidkarlsen]

More stuff, seems like a UI-glitch:

If you pull:

docker pull --platform linux/amd64 MYREG/docker.io/library/ubuntu:noble

it will actually download, but it won't show when you list repositories in the azure portal. Others that are non-library ones, for instance docker.io/bitnami/sealed-secrets-controller will be listed...

[luisdlp]

@davidkarlsen are you using anonymous pulls from DockerHub? Starting on 6/30, DH started enforcing normal rate limiting rates for Azure customers. We highly recommend getting a free or paid subscription so that you have a higher rate limit. Caching does an initial import of the image into the ACR and then pulls it from there. If the import fail (likely due to throttling) the image will be pulled directly from DH. Since the import is not succeeding, you are not able to see the image cached in the registry. Can you try with authenticated pull?

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity.