Azure
`az functionapp keys list` throws 'Bad Request'
Describe the bug
Command Name
az functionapp keys list
Errors:
Operation returned an invalid status 'Bad Request'
The app name and resource group are correct since az functionapp show -g {} -n {} with the same values works.
To Reproduce:
- Linux Function App v4.
- Configured to pull container image from Azure Container Registry using managed identity.
- Storage Account has been created with
--public-network-access Disabled. -
az functionapp keys list -g {} -n {}
Expected Behavior
- Lists the keys (or shows an empty list like I would've expected in this particular case) or
- more informative error message.
Environment Summary
Linux-4.4.0-19041-Microsoft-x86_64-with-debian-bullseye-sid, Ubuntu 20.04 LTS
Python 3.6.10
Installer: DEB
azure-cli 2.31.0
Extensions:
aks-preview 0.5.49
Additional Context
Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @ahmedelnably, @fabiocav.
Issue Details
Describe the bug
Command Name
az functionapp keys list
Errors:
Operation returned an invalid status 'Bad Request'
The app name and resource group are correct since az functionapp show -g {} -n {} with the same values works.
To Reproduce:
- Linux Function App v4.
- Configured to pull container image from Azure Container Registry using managed identity.
-
az functionapp keys list -g {} -n {}
Expected Behavior
- Lists the keys (or shows an empty list like I would've expected in this particular case) or
- more informative error message.
Environment Summary
Linux-4.4.0-19041-Microsoft-x86_64-with-debian-bullseye-sid, Ubuntu 20.04 LTS
Python 3.6.10
Installer: DEB
azure-cli 2.31.0
Extensions:
aks-preview 0.5.49
Additional Context
| Author: | kimjamia |
|---|---|
| Assignees: | - |
| Labels: |
|
| Milestone: | - |
![msftbot[bot] avatar](https://avatars.githubusercontent.com/in/26612?v=4 "Published by a pub.dev verified publisher"){kind=small}
I updated the steps to reproduce. This is most likely related to the fact that the storage account was created with --public-network-access Disabled. However, when trying to create a minimal setup that reproduces this issue, I couldn't get it to throw Bad Request. Instead it returns:
(NotFound) Error retrieving master key. <html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.21.5</center>
</body>
</html>
Code: NotFound
Message: Error retrieving master key. <html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.21.5</center>
</body>
</html>
Our deployment pipeline retrieves the Function App master key to invoke a HTTP trigger as part of an automated smoke-test after deployment of our application. We notice that this step sometimes randomly fails with a "Bad Request", and when we try to retrieve the key a few minutes later it succeeds.
This is the command:
az functionapp keys list -g "xxxxx" -n "xxxxx" --query "masterKey" --output tsv
This is the error reported by 'az':
ERROR: Operation returned an invalid status 'Bad Request'
And the Activity Log on the Function App contains an entry similar to this:
{
"eventName": {
"value": "EndRequest",
"localizedValue": "End request"
},
"level": "Error",
"operationName": {
"value": "Microsoft.Web/sites/host/listkeys/action",
"localizedValue": "List Web Apps Functions Host Keys"
},
"resourceId": "/subscriptions/xxxxxxxx/resourceGroups/xxxxxxxx/providers/Microsoft.Web/sites/xxxxxxxx/host/default",
"status": {
"value": "Failed",
"localizedValue": "Failed"
},
"subStatus": {
"value": "BadRequest",
"localizedValue": "Bad Request (HTTP Status Code: 400)"
},
"properties": {
"statusCode": "BadRequest",
"serviceRequestId": null,
"statusMessage": "{\"Code\":\"BadRequest\",\"Message\":\"Encountered an error (InternalServerError) from host runtime.\",\"Target\":null,\"Details\":[{\"Message\":\"Encountered an error (InternalServerError) from host runtime.\"},{\"Code\":\"BadRequest\"},{\"ErrorEntity\":{\"Code\":\"BadRequest\",\"Message\":\"Encountered an error (InternalServerError) from host runtime.\"}}],\"Innererror\":null}",
"eventCategory": "Administrative",
"message": "Microsoft.Web/sites/host/listkeys/action",
}
}
I'm not sure what the previously mentioned --public-network-access option does and I don't think it's related to our issue, but for completeness sake I'll mention that our storage account is created via an ARM template incremental deployment with the following properties:
"properties": {
"allowBlobPublicAccess": false,
"minimumTlsVersion": "TLS1_2",
"supportsHttpsTrafficOnly": true,
"networkAcls": {
"bypass": "AzureServices",
"virtualNetworkRules": [
{
"id": "[resourceId('Microsoft.Network/virtualNetworks/subnets', variables('vnet-name'), variables('subnet-functionapp-name'))]",
"action": "Allow"
}
],
"ipRules": [],
"defaultAction": "Deny"
}
},
Can we get an update on this please. I've been dealing with this issue for several months now and it seems very hit or miss as to whether this az cli request works or not. If it was broken then I'd be more understanding but the variability of the request succeeding is most frustrating. Also, I've tried with both public network access and allow blob public access enabled and I still get this issue:
Adding some debug logs from the az cli call:
az functionapp keys list -n myFunction -g myResourceGroup --debug
FYI, this is failing in the Azure Portal also for certain functions that have function key interactions:
hey all - it would be best to open a support case for these - since this is not specific to CLI. I did try to report these to the function apps team directly, but not something we can take action via CLI.
Yes would be best to open a support ticket or you can transfer this issue to the Functions Host repo: https://github.com/Azure/azure-functions-host/issues
Az CLi was onboarded to CXP from Feb 1st. Removing the CXP Attention label from this issue considering this issue was created in Jan. CC @yonzhan
Is there any update? Steps to reproduce:
az storage account create --name $(StorageAccountName) --resource-group $(StorageResourceGroupName)
az appservice plan create --name $(AppServicePlanName) --resource-group $(AppServicePlanResourceGroupName) --sku $(AppServicePlanSku) --location $(AppServicePlanLocation)
az functionapp create --resource-group $(FunctionResourceGroupName) --plan $(AppServicePlanPath) --name $(FunctionName) --storage-account $(StorageAccountPath) --functions-version $(FunctionVersion) --os-type $(FunctionOs) --runtime dotnet --disable-app-insights true --app-insights-key $(ApplicationInsightsImbasKey) --subnet $(FunctionSubnetPath)
az functionapp keys list --name $(FunctionName) --resource-group $(FunctionResourceGroupName)
Last command returns: Operation returned an invalid status 'Bad Request'
az rest command returns:
az rest --method post --uri "/subscriptions/xyz/resourceGroups/rg-func/providers/Microsoft.Web/sites/func-test/host/default/listKeys?api-version=2022-03-01" --query functionKeys.default --output tsv
Bad Request({"Code":"BadRequest","Message":"Encountered an error (InternalServerError) from host runtime.","Target":null,"Details":[{"Message":"Encountered an error (InternalServerError) from host runtime."},{"Code":"BadRequest"},{"ErrorEntity":{"Code":"BadRequest","Message":"Encountered an error (InternalServerError) from host runtime."}}],"Innererror":null})
Also in the Azure Portal the App Key are not shown and cannot be set:
Do I miss something how to create a function using azure cli? How can I add a key to my function?
Is there any update? Steps to reproduce:
az storage account create --name $(StorageAccountName) --resource-group $(StorageResourceGroupName) az appservice plan create --name $(AppServicePlanName) --resource-group $(AppServicePlanResourceGroupName) --sku $(AppServicePlanSku) --location $(AppServicePlanLocation) az functionapp create --resource-group $(FunctionResourceGroupName) --plan $(AppServicePlanPath) --name $(FunctionName) --storage-account $(StorageAccountPath) --functions-version $(FunctionVersion) --os-type $(FunctionOs) --runtime dotnet --disable-app-insights true --app-insights-key $(ApplicationInsightsImbasKey) --subnet $(FunctionSubnetPath) az functionapp keys list --name $(FunctionName) --resource-group $(FunctionResourceGroupName)
Last command returns: Operation returned an invalid status 'Bad Request'
az rest command returns:
az rest --method post --uri "/subscriptions/xyz/resourceGroups/rg-func/providers/Microsoft.Web/sites/func-test/host/default/listKeys?api-version=2022-03-01" --query functionKeys.default --output tsv
Bad Request({"Code":"BadRequest","Message":"Encountered an error (InternalServerError) from host runtime.","Target":null,"Details":[{"Message":"Encountered an error (InternalServerError) from host runtime."},{"Code":"BadRequest"},{"ErrorEntity":{"Code":"BadRequest","Message":"Encountered an error (InternalServerError) from host runtime."}}],"Innererror":null})
Also in the Azure Portal the App Key are not shown and cannot be set:
Do I miss something how to create a function using azure cli? How can I add a key to my function?
@MrMeterPing unfortunately, since this is not a CLI client specific problem but rather API/runtime please use the link above to report the issue https://github.com/Azure/azure-functions-host/issues & some one from functions team can help move this forward. //CC: @anirudhgarg
I ran into this both with CLI (az functionapp keys list/set) and Azure portal. I finally figured out how to work around it.
Initial setup:
- Azure Function with VNet integration and vnetRouteAllEnabled = true --- Environment had UDR that routed all VNet outgoing traffic through an NVA
- Storage account associated with Function had public access disabled
Symptoms: portal didn't show any keys, and attempts to set a key got BadRequest.
What I eventually found worked:
- On the Function, set vnetRouteAllEnabled = false
- Set the storage account to allow public access; in my case, I added network access rules for my IP range and left defaultAction = Deny - in the portal this corresponds to "Enabled from selected virtual networks and IP addresses" so that the storage account isn't wide open
Once I did this, listing and setting keys worked. I was able to repeatedly switch back and forth between fail (Function had vnetRouteAllEnabled = true and/or storage account had public access disabled) and the above success conditions.
On the Function side, this makes sense where I am working from outside the VNet - if all the Function traffic is set to route through the VNet and there is a UDR to route outgoing traffic through an NVA, and the NVA "swallows" the return to me, no wonder I get a timeout.
But I wasn't able to find this dependency on storage account access documented anywhere. Maybe I missed it, but if not, would be good to document it as this cost me hours of debugging. On that note, the BadRequest error messages contain no detail whatsoever, any hint there of "the storage account isn't configured as needed" would have been helpful.
