2.61.0 `az webapp deploy` for zip deploy polls for status of async deployment and never finishes

[gburkecw]

Describe the bug

Yesterday, on azcli 2.60.0, deploying to a Linux web app with a zip deploy package worked fine and had the following output:

Today on azcli 2.61.0, the command polls forever and never exits, even though the Kudu portal shows that the deployment completed:

Notice the new "polling the status of async deployment". I tried heeding the deprecation warning and switching to use az webapp deploy and I set --async true, but I still get the message "polling the status of async deployment" which never finishes:

Related command

az webapp deployment source config-zip --name "" --resource-group "" --slot "" --src "";

also tried (due to deprecation warning), tried async=true to try to get rid of polling:

az webapp deploy --async true --name "" --resource-group "" --slot "" --src-path "" --type zip;

Errors

ERROR: Timeout reached while tracking deployment status, however, the deployment operation is still on-going.

Issue script & Debug output

WARNING: cli.azure.cli.command_modules.appservice.custom: Status: Build successful. Time: 78(s) DEBUG: cli.azure.cli.core.util: Found subscription ID b731a80f-7771-4552-b828-0508b3e54da8 in the URL https://management.azure.com/subscriptions/b731a80f-7771-4552-b828-0508b3e54da8/resourceGroups/rg-webapp-api-pr-eus/providers/Microsoft.Web/sites/cwappapipreus/slots/Deploying/deploymentStatus/2b7fbb1c-7c22-4ff8-9df1-d4bb9ae34a75?api-version=2023-01-01 DEBUG: cli.azure.cli.core.util: Retrieving token for resource https://management.core.windows.net/, subscription b731a80f-7771-4552-b828-0508b3e54da8 DEBUG: urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: msal.authority: Initializing with Entra authority: https://login.microsoftonline.com/*** DEBUG: msal.authority: openid_config("https://login.microsoftonline.com//v2.0/.well-known/openid-configuration") = {'token_endpoint': 'https://login.microsoftonline.com//oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com//discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com//v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com//oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com//oauth2/v2.0 DEBUG: msal.application: Broker enabled? None DEBUG: cli.azure.cli.core.auth.msal_authentication: ServicePrincipalCredential.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={} DEBUG: msal.application: Cache hit an AT DEBUG: msal.telemetry: Generate or reuse correlation_id: 0e47d629-b568-48fa-a28c-a230e5aa6750 INFO: cli.azure.cli.core.util: Request URL: 'https://management.azure.com/subscriptions/b731a80f-7771-4552-b828-0508b3e54da8/resourceGroups/rg-webapp-api-pr-eus/providers/Microsoft.Web/sites/cwappapipreus/slots/Deploying/deploymentStatus/2b7fbb1c-7c22-4ff8-9df1-d4bb9ae34a75?api-version=2023-01-01' INFO: cli.azure.cli.core.util: Request method: 'GET' INFO: cli.azure.cli.core.util: Request headers: INFO: cli.azure.cli.core.util: 'User-Agent': 'python/3.11.8 (Linux-5.10.102.2-microsoft-standard-x86_64-with-glibc2.35) AZURECLI/2.61.0 (DEB)' INFO: cli.azure.cli.core.util: 'Accept-Encoding': 'gzip, deflate' INFO: cli.azure.cli.core.util: 'Accept': '/' INFO: cli.azure.cli.core.util: 'Connection': 'keep-alive' INFO: cli.azure.cli.core.util: 'x-ms-client-request-id': 'dcabce40-3f4f-4739-8e31-d63076b08ac0' INFO: cli.azure.cli.core.util: 'CommandName': 'webapp deploy' INFO: cli.azure.cli.core.util: 'ParameterSetName': '--async --name --resource-group --slot --src-path --type --debug' INFO: cli.azure.cli.core.util: 'Authorization': '***' INFO: cli.azure.cli.core.util: Request body: INFO: cli.azure.cli.core.util: None DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443 DEBUG: urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions/b731a80f-7771-4552-b828-0508b3e54da8/resourceGroups/rg-webapp-api-pr-eus/providers/Microsoft.Web/sites/cwappapipreus/slots/Deploying/deploymentStatus/2b7fbb1c-7c22-4ff8-9df1-d4bb9ae34a75?api-version=2023-01-01 HTTP/1.1" 202 568 INFO: cli.azure.cli.core.util: Response status: 202 INFO: cli.azure.cli.core.util: Response headers: INFO: cli.azure.cli.core.util: 'Cache-Control': 'no-cache' INFO: cli.azure.cli.core.util: 'Pragma': 'no-cache' INFO: cli.azure.cli.core.util: 'Content-Length': '568' INFO: cli.azure.cli.core.util: 'Content-Type': 'application/json' INFO: cli.azure.cli.core.util: 'Expires': '-1' INFO: cli.azure.cli.core.util: 'ETag': '"1DAABB9DCD07780"' INFO: cli.azure.cli.core.util: 'Location': 'https://management.azure.com/subscriptions/b731a80f-7771-4552-b828-0508b3e54da8/resourceGroups/rg-webapp-api-pr-eus/providers/Microsoft.Web/sites/cwappapipreus/slots/Deploying/deploymentStatus/2b7fbb1c-7c22-4ff8-9df1-d4bb9ae34a75?api-version=2023-01-01&t=638519186759709296&c=MIIHhzCCBm-gAwIBAgITfATUDnhanTLRPeyZxwAABNQOeDANBgkqhkiG9w0BAQsFADBEMRMwEQYKCZImiZPyLGQBGRYDR0JMMRMwEQYKCZImiZPyLGQBGRYDQU1FMRgwFgYDVQQDEw9BTUUgSW5mcmEgQ0EgMDUwHhcNMjQwNTA5MTgwOTM3WhcNMjUwNTA0MTgwOTM3WjBAMT4wPAYDVQQDEzVhc3luY29wZXJhdGlvbnNpZ25pbmdjZXJ0aWZpY2F0ZS5tYW5hZ2VtZW50LmF6dXJlLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAL9dgMYYraJFlILLieCu28kujI4Eh_o_VNUMODlVyQkeY76IALT8f-X5D0AbgZujawag12jT_t6b5XuEUL5d-K_TBnvCQbYD9roAD-qWL6Aw7DznLoWvMoAX4tn8ngoKup3mxnKUPJ9c09dr_CTsE6QjpcT4DYzTZ0jQ_Nsi7kY20vZZLm1FsTuPzkG8N4XnapOt6O2JPSgdtRa68z-kHJia0Z0zfCVBpqN4T4a_depQa2MOOXTmsPmm6J4GRhj6UdTJZ1TyBGGI3MmjNY6ih6M6NilZJWZuQXuNG1LQ_PonDW8K_rfK2zzobonkASWw2_T1KI_w2cZ2zfPKmQ0gmgECAwEAAaOCBHQwggRwMCcGCSsG INFO: cli.azure.cli.core.util: 'Retry-After': '15' INFO: cli.azure.cli.core.util: 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains' INFO: cli.azure.cli.core.util: 'X-AspNet-Version': '4.0.30319' INFO: cli.azure.cli.core.util: 'X-Powered-By': 'ASP.NET' INFO: cli.azure.cli.core.util: 'x-ms-ratelimit-remaining-subscription-reads': '11999' INFO: cli.azure.cli.core.util: 'x-ms-request-id': '4fa9a185-bad3-45bf-adf4-f4e1146fcdf1' INFO: cli.azure.cli.core.util: 'x-ms-correlation-request-id': '4fa9a185-bad3-45bf-adf4-f4e1146fcdf1' INFO: cli.azure.cli.core.util: 'x-ms-routing-request-id': 'EASTUS:20240521T200435Z:4fa9a185-bad3-45bf-adf4-f4e1146fcdf1' INFO: cli.azure.cli.core.util: 'X-Content-Type-Options': 'nosniff' INFO: cli.azure.cli.core.util: 'X-Cache': 'CONFIG_NOCACHE' INFO: cli.azure.cli.core.util: 'X-MSEdge-Ref': 'Ref A: 60B6B503A30549908BD08B3D1F8FE146 Ref B: BL2AA2030104003 Ref C: 2024-05-21T20:04:35Z' INFO: cli.azure.cli.core.util: 'Date': 'Tue, 21 May 2024 20:04:35 GMT' INFO: cli.azure.cli.core.util: Response content: INFO: cli.azure.cli.core.util: {"id":"/subscriptions/b731a80f-7771-4552-b828-0508b3e54da8/resourceGroups/rg-webapp-api-pr-eus/providers/Microsoft.Web/sites/cwappapipreus/slots/Deploying/deploymentStatus/2b7fbb1c-7c22-4ff8-9df1-d4bb9ae34a75","name":"2b7fbb1c-7c22-4ff8-9df1-d4bb9ae34a75","type":"Microsoft.Web/sites/slots/deploymentStatus","location":"East US","tags":{},"properties":{"deploymentId":"2b7fbb1c-7c22-4ff8-9df1-d4bb9ae34a75","status":"BuildSuccessful","numberOfInstancesInProgress":0,"numberOfInstancesSuccessful":0,"numberOfInstancesFailed":0,"failedInstancesLogs":null,"errors":null}} WARNING: cli.azure.cli.command_modules.appservice.custom: Status: Build successful. Time: 94(s)

Expected behavior

Either the deprecated az webapp deployment source config-zip command should continue to work as it did in 2.60.0 (returns successful deployment immediately) or the az webapp deploy command should exhibit the documented behavior of --async true and return right away.

Also, since the deployment is successful, I would think that --async false would also work fine in this case.

Environment Summary

azure-cli 2.61.0

core 2.61.0 telemetry 1.1.0

Dependencies: msal 1.28.0 azure-mgmt-resource 23.1.1

Python location '/opt/az/bin/python3' Extensions directory '/root/.azure/cliextensions'

Python (Linux) 3.11.8 (main, May 16 2024, 03:47:28) [GCC 11.4.0]

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[gburkecw]

I think what is probably happening here is that 2.61.0 uses api-version=2023-01-01 for the deploymentStatus API, and it is returning 202 instead of 200. Perhaps the Azure CLI is looking for a 200 result specifically. I confirmed with an older version of Azure CLI (2.43) that it is using an older version of the deploymentStatus API which is returning 200 instead of 202.

[btardif]

@tulikac can you take a look, this might be related to deployment status?

[mderriey]

We're facing the same issue on two separate pipelines running on GitHub Actions.

The new GitHub Actions runner image for Ubuntu 22.04 was recently updated from 20240516.1 to 20240526.1 which bumped the Azure CLI from v2.60.0 to v2.61.0.

We'll investigate manually installing v2.60.0 at the start of the job to work around this.

[cjhannon84]

@mderriey We also faced this issue, I downgraded the cli version we are using and all's good while we await a fix in the latest version.

[mderriey]

Here's the step we use in our GitHub Actions deployment jobs running on ubuntu-latest until this issue is fixed:

- name: Install Azure CLI v2.60.0
  shell: bash
  run: |
    AZ_DIST=$(lsb_release -cs)
    AZ_VER=2.60.0
    sudo apt-get install -y --allow-downgrades azure-cli=${AZ_VER}-1~${AZ_DIST}

[cijalba]

This is the step we use in our BitBucket as workaround too:

    - step: &deployment
        name: "Deploy to Azure"
        image: mcr.microsoft.com/azure-cli:2.60.0
        deployment: STG
        script:
          - az login --service-principal ...
          - az webapp deploy --subscription $AZURE_SUBSCRIPTION --resource-group $AZURE_GROUP --name $AZURE_SERVICE_NAME --src-path $BITBUCKET_BUILD_NUMBER.zip --type zip --async true

[GranPC]

If I had to take a wild guess, I'd assume 18c440fc340bbf3f38513469dbc8c3d45c4cadcc is the commit that broke it. Azure, please fix :pray:

[RobVeneberg2]

You can also use the --track-status flag to disable status tracking (instead of downgrading):

--track-status false

[EugeneDeveloper]

You can also use the --track-status flag to disable status tracking (instead of downgrading):

--track-status false

Unfortunately, command ignores this argument. It still pulls status permanently.

[RobVeneberg2]

@EugeneDeveloper Strange, it works for us. Are you using version 2.61? The PR describes this flag: https://github.com/Azure/azure-cli/pull/28921

[cijalba]

You can also use the --track-status flag to disable status tracking (instead of downgrading):

--track-status false

--track-status If true, web app startup status during deployment will be tracked for linux web apps.

accepted values: false, true default value: True

Probably it doesn't work for windows web apps, only for linux ones...

[tinytim84]

You can also use the --track-status flag to disable status tracking (instead of downgrading): --track-status false

--track-status If true, web app startup status during deployment will be tracked for linux web apps.

accepted values: false, true default value: True

Probably it doesn't work for windows web apps, only for linux ones...

This works for me on a Linux build.

[cijalba]

You can also use the --track-status flag to disable status tracking (instead of downgrading): --track-status false

--track-status If true, web app startup status during deployment will be tracked for linux web apps.

accepted values: false, true default value: True

Probably it doesn't work for windows web apps, only for linux ones...

Verified on the python code:

Option enabled for Linux only

@EugeneDeveloper you should have this errors on your logs:

"Deployment status tracking is currently only supported for linux webapps." " Resuming without tracking status."

[EugeneDeveloper]

It works fine with the latest version (2.61.0). Thanks a lot for the tip.

[denis-taran]

I have the same problem. Linux web applications could not be deployed properly without the workaround.

[breggles]

Same here. Linux, az version 2.62.

[lyager]

Still a problem - using the AZ the comes with the Ubuntu runner.

[sarsharma]

Hi all, the issue with the deployment status being stuck on "Build Successful" is due to a bug we discovered with deployment status API when trying to deploy to slots. We had disabled use of deployment status API for slots starting with az-cli version 2.63.0 and you should not see this issue if you update CLI to the latest version We are actively working on fixing the issue with the API and adding back support for it when deploying to slots. Deploying to your main app slot still makes use of the deployment status API and should work as intended. If you are trying to deploy to a slot and are unable to upgrade your CLI, you can pass --track-status false as an additional param in your deployment command. Please let us know if the issue persists. Apologies for the inconvenience.

[glenthomas]

If you are trying to deploy to a slot and are unable to upgrade your CLI, you can pass --track-status false as an additional param in your deployment command. Please let us know if the issue persists. Apologies for the inconvenience.

@sarsharma if we are deploying to a slot, how do we know when all of the instances have finished deploying?

[alexkunde]

We are using 2.67 from the Github Runner image ubuntu-latest and we still see this behaviour. We had to implement the fix.

[glenthomas]

I think the only real fix is to stop using Azure App Services. Too many issues.

[danladis]

Is there any update on this? We face the same issue in our bitbucket pipeline No matter if --track-status false or --async true

[TArtiushenko]

same issue, running pipeline in azure devops, AzureCLI@2 (yes i need it exactly using az cli not built in task) 2.80.0