Azure
Improve documentation surrounding how function keys are managed when AzureWebJobsSecretStorageType=kubernetes
First, I will refer you to this question I put on the microsoft community help fora a couple days ago.
Please read ^^^^ this question ^^^^ thoroughly as well as the answer I reported earlier today to get a good grip on the background of this request, and to understand how I ultimately found the solution.
The primary documentation I found on how to manage function app secrets in k8s was on this and this Microsoft Learn page.
It took me forever to find an example k8s manifest for how to declare the secret, but I eventually found one in a description of a PR here. I used this to come up with this manifest for my own application
apiVersion: v1
data:
function.HttpTrigger.default: ++++++++
host.master: ++++++++
kind: Secret
immutable: true
metadata:
annotations:
ecosystem: dotnet
owning-team: vrbe
pillar: VR
runtime: Azure functions host runtime in docker
creationTimestamp: '2025-03-25T18:57:09Z'
labels:
app: vrbe-auth
argocd.argoproj.io/instance: vrbe-auth
helm.sh/chart: common-1.0.0
name: vrbe-auth-fn-keys
namespace: vrbe
resourceVersion: '336197458'
uid: 26460870-c4a2-409c-aab6-0ea21fdd5d3b
type: Opaque
I can't say for sure exactly what was going on and what was causing that 500 since I have no context inside the transport that modifies the k8s secret or that routes requests from the http entry down to the function, but my best guess is that my application runtime was unable to update this k8s secret, and therefore, the application container was facing errors.
There are a few problems with this:
- I would expect the container runtime to report problems interacting with k8s in the form of a load of error logs if it cannot modify the a kubernetes secret. There was no visibility into what the problem even was by reviewing application logs. There was simply nothing there.
- If there is a logging configuration one can set up in order to get these logs to be reported to the console, then the logging documentation makes no mention of it.
- There is no documentation inside of the linked Microsoft Learn documents as to what the
dataof the secret should be. I pulledhost.masterandfunction.HttpTrigger.defaultfrom a 6-year-old PR, and it magically seemed to work. There needs to be improved documentation about what would constitute well-formed k8s secret data, not to mention a huge warning that theimmutableflag on the manifest cannot be set.
@earlofhemsley
I to have been trying get around the 500 error issue and can't seem to figure out how exactly this is supposed to work.
Sorry, I am new to running azure functions on k8s. Trying to figure out how exactly the deployment will find the secret and where it will mount.
Do you mind sharing a sampledeployment.yaml and host.json or local.settings.json?
additional context, I am just using kubectl apply in my local k8s to test it
I also noticed this comment which I am guessing could work as well https://github.com/Azure/azure-functions-host/issues/4147#issuecomment-477442831
@paravatha thanks for responding!
I took a best guess based on this documentation (previously linked as well) and this seems to be all you need to do in the deployment configuration to make the container try to find a k8s secret to use as its function key store
apiVersion: apps/v1
kind: Deployment
metadata: {
# ... omitted ...
}
spec:
# ... metadata, other things ...
template:
spec:
containers:
# ... image, other things ...
- env:
- name: AzureWebJobsSecretStorageType
value: kubernetes
- name: AzureWebJobsKubernetesSecretName
value: vrbe-auth-fn-keys
# many other env vars
status: {
# ... omitted ...
}
here's the live (read: working) manifest for the k8s secret vrbe-auth-fn-keys
apiVersion: v1
data:
functions.userauthredirect.default: ++++++++
host.master: ++++++++
kind: Secret
metadata:
annotations:
ecosystem: dotnet
owning-team: vrbe
pillar: VR
runtime: Azure functions host runtime in docker
creationTimestamp: '2025-03-25T18:57:09Z'
labels:
app: vrbe-auth
argocd.argoproj.io/instance: vrbe-auth
helm.sh/chart: common-1.0.0
name: vrbe-auth-fn-keys
namespace: vrbe
resourceVersion: '336197458'
uid: 26460870-c4a2-409c-aab6-0ea21fdd5d3b
type: Opaque
that appears to be good enough, at least in my case, to get the function app to reach out to the secret for function key management.
I also noticed this comment which I am guessing could work as well https://github.com/Azure/azure-functions-host/issues/4147#issuecomment-477442831
if i hadn't been able to make the k8s secret work like I did, I probably would have pursued mounting a directory in my container runtime containing a file that the function app runtime could manipulate, similar to what's been done in that linked issue. I didn't get to that point, though.
@earlofhemsley thanks for the quick reply. I got it working with mount path this morning, but not with this approach you tried.
- name: AzureWebJobsSecretStorageType
value: kubernetes
- name: AzureWebJobsKubernetesSecretName
value: vrbe-auth-fn-keys
Maybe, I messed up my config, Will try again and report back
@earlofhemsley works now. Thanks a lot!!!
One additional thing I had to do in my setup as I didn't have it.
the service account used by the pod needs these permissions verbs: ["get", "list", "patch", "update"] on secrets
not verbs: ["create","list","patch","update","get","list"] mentioned here https://github.com/Azure/azure-functions-host/issues/9049
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: az-func-api-role
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "patch", "update"]
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: az-func-api-rolebinding
subjects:
- kind: ServiceAccount
name: az-func-api-sa
roleRef:
kind: Role
name: az-func-api-role
apiGroup: rbac.authorization.k8s.io
so, I had to create role and role-binding for that service account
so, I had to create role and role-binding for that service account
sounds like this would be excellent information to include in whatever improved documentation comes of this. Thanks @paravatha
@earlofhemsley did you ever run into this 500 internal issue mentioned here?
https://github.com/Azure/azure-functions-host/issues/10740
Did you see 500 errors when you enabled detailed logging ?
@earlofhemsley did you ever run into this
500 internalissue mentioned here? Azure/azure-functions-host#10740
That sounds very close to my issue, except that after i removed the immutable attribute from the secret, the azure functions runtime reached out to the secret and modified it to place a function key for the route that I had just called. While it was immutable, though, that is the behavior that I observed.
Did you see 500 errors when you enabled detailed logging ?
Detailed logging whaaaaat? If there's a way to enable detailed logging that I didn't already do, please tell me what that is and how. I configured logging down to Trace for everything I knew how to configure. See the original microsoft community help post I made for more info on how I configured logging.
Detailed logging whaaaaat? If there's a way to enable detailed logging that I didn't already do, please tell me what that is and how. I configured logging down to Trace for everything I knew how to configure. See the original microsoft community help post I made for more info on how I configured logging.
Apologies, I misread/mis-understood your comment above. Enabling different levels as below didn't provide any insights in to the 500 Internal Server Error. Thanks again for your help!
"logging": {
"logLevel": {
"default": "Trace",
"Function": "Trace",
"Host.Results": "Trace",
"Host.Aggregator": "Trace",
"Azure.Core": "Trace",
"Microsoft": "Trace",
"Worker": "Trace"
},
Documenting different options for posterity for anyone running into similar issues References: https://learn.microsoft.com/en-us/azure/azure-functions/function-keys-how-to?tabs=azure-portal#manage-key-storage https://www.github.com/Azure/azure-functions-host/pull/4462#issue-445268179
Option 1 - This requires additional permissions to the service account, the app creates api keys on the fly.
Downside: not a good idea due to Enforcing least privilege access in Kubernetes environments pattern.
this doesn't make sense to me, bit of a head scratcher
role.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: az-func-api-role
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create","list","patch","update","get","list"]
role-binding.yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: az-func-api-rolebinding
subjects:
- kind: ServiceAccount
name: az-func-api-sa
roleRef:
kind: Role
name: az-func-api-role
apiGroup: rbac.authorization.k8s.io
part of deployment.yaml
serviceAccountName: az-func-api-sa
containers:
- name: api-container
image: registry.azurecr.io/az-func-api:test
env:
- name: AzureWebJobsSecretStorageType
value: kubernetes
- name: AzureWebJobsKubernetesSecretName
value: az-func-api-keys
Option 2 - mount api keys as a file.
Downside: not a good idea if you are using a key vault lke hashicorp vault, need to maintain api keys in 2 places (1st as a file for the api server app, 2nd as a key to the client app). AzureWebJobsSecretStorageType:files is not recommended as its an old pattern https://learn.microsoft.com/en-us/azure/azure-functions/function-keys-how-to?tabs=azure-portal#manage-key-storage
secrets.yaml
apiVersion: v1
kind: Secret
metadata:
name: az-func-api-keys
type: Opaque
stringData:
host.json: |-
{
"masterKey": {
"name": "master",
"value": "masterkey",
"encrypted": false
},
"functionKeys": [ ]
}
part of deployment.yaml
env:
- name: AzureWebJobsSecretStorageType
value: files
volumeMounts:
- name: functions-keys
mountPath: "/azure-functions-host/Secrets"
readOnly: true
volumes:
- name: functions-keys
secret:
secretName: az-func-api-keys
Option 3 - mount api keys as key-value pairs
This is a kind of a hybrid approach, which doesnt require additional privelges and api keys can be maintained in a single place in the vault, and uses AzureWebJobsSecretStorageType: kubernetes approach
secrets.yaml
apiVersion: v1
kind: Secret
metadata:
name: az-func-api-keys
type: Opaque
data:
host.master: base64encodedkey
part of deployment.yaml
env:
- name: AzureWebJobsSecretStorageType
value: kubernetes
volumeMounts:
- mountPath: /run/secrets/functions-keys
name: functions-keys
readOnly: true
volumes:
- name: functions-keys
secret:
secretName: az-func-api-keys