Azure
Deployment to AKS uses old image
I'm trying to update an endpoint, that is already deployed at AKS. Here is my workflow:
- I'm updating a custom image - e.g. update installed libraries. It is registered in private ACR.
- I call environment.build() - forcing an azureml image to be rebuilt, so that changes in source image will be updated in final image. Please notice, that the built image has the same name:
d2bbe17668204c4f957f17cca62a7db4.azurecr.io/azureml/azureml_02fba29521dca17addff8c3b5f3d93cb. The only thing that changes is the source docker, which is reflected with a new manifest. - Deployment of the new model:
webservice = Model.deploy( workspace=ws, name=service_name, models=[model], inference_config=inference_config, deployment_config=deployment_config, show_output=True, overwrite=True )
Deployment goes well, however the app runs on the previous image.
In the ACR the latest version is:
d2bbe17668204c4f957f17cca62a7db4.azurecr.io/azureml/azureml_02fba29521dca17addff8c3b5f3d93cb:latest
and sha256:19b9c6dfaa3cd0ad4364719fcdf2a170606b9e43106225748f6785291c43d7f4
while if I describe a pod in K8 it uses the previous version from cache:
d2bbe17668204c4f957f17cca62a7db4.azurecr.io/azureml/azureml_02fba29521dca17addff8c3b5f3d93cb@sha256:bc7911cd9109ea3748feba3a56b1f8f8bc8c65b154d75a54bf96b87d80b5e472
I think that's due to imagePullPolicy: IfNotPresent.
In the kubernetes describe pod I can see, that an image was not pulled but used from cache instead.
I couldn't find any way to change this setting to Always.
I have patched the deployment map and correct image has been pulled, so I know that this option ("Always") would fix this issue for me. Thanks for any help!
Did you specify user-managed dependencies? If yes, then conda dependencies are omitted. So the built image will be same. https://docs.microsoft.com/en-us/azure/machine-learning/concept-environments#building-environments-as-docker-images
Yes, I have specified user-managed dependencies. However when the base image is changed, eg. some libraries were updated, a new build is registered in the acr. And, due to pull policy the old version is used for the pod on k8 instead of the newest one. For now I made a workaround - I've added a versioning system to image tags, so that azure is forced to use the latest image. But I think that it should work for the "latest" tag as well.
Hi,
spec: containers:
- name: your-container-name image: d2bbe17668204c4f957f17cca62a7db4.azurecr.io/azureml/azureml_02fba29521dca17addff8c3b5f3d93cb:latest imagePullPolicy: Always
Let me know if this resolves your issue or if you have any further questions!
Best regards, Talmeez Fuaad
