sops icon indicating copy to clipboard operation
sops copied to clipboard

ForbiddenByRbac when using azure key vault backend with version 3.8+

Open andrey-gava opened this issue 1 year ago • 7 comments

Hi! We successfully using sops 3.7.3 with azure key vault as backend. But when we try to use same flow with 3.8+ version it fails with ForbiddenByRbac error. I tried both login type - az login and service principle credentials. Both fails. I have next roles permission to resource: [Key Vault Crypto Officer, Key Vault Crypto User]

Something changed in how sops authenticate with azure resources?

./sops-v3.8.1.linux.amd64 ~/git/environments/aks-saas/secrets.yaml

Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  https://*******.vault.azure.net/keys/sops-aks-saas-key/*********************: FAILED
    - | failed to decrypt sops data key with Azure Key Vault key
      | 'https://*******.vault.azure.net/keys/sops-aks-saas-key/*********************':
      | POST
      | https://*******.vault.azure.net/keys/sops-aks-saas-key/*********************/decrypt
      | --------------------------------------------------------------------------------
      | RESPONSE 403: 403 Forbidden
      | ERROR CODE: Forbidden
      | --------------------------------------------------------------------------------
      | {
      |   "error": {
      |     "code": "Forbidden",
      |     "message": "Caller is not authorized to perform action
      | on resource.\r\nIf role assignments, deny assignments or
      | role definitions were changed recently, please observe
      | propagation time.\r\nCaller:
      | appid=********************;oid=*********************;iss=https://sts.windows.net/**************/\r\nAction:
      | 'Microsoft.KeyVault/vaults/keys/decrypt/action'\r\nResource:
      | '/subscriptions/************************/resourcegroups/****/providers/microsoft.keyvault/vaults/*******/keys/sops-aks-saas-key'\r\nAssignment:
      | (not found)\r\nDenyAssignmentId: null\r\nDecisionReason:
      | null \r\nVault: *******;location=********\r\n",
      |     "innererror": {
      |       "code": "ForbiddenByRbac"
      |     }
      |   }
      | }
      | --------------------------------------------------------------------------------

andrey-gava avatar Apr 16 '24 08:04 andrey-gava

I know this is an old post but... Where is this command being launched? I had the same issue on our self-hosted gh agents when updated to sops latest version and had to rollback to ensure this was working. In my case, the OS was authenticating to the keyvault using the managed identity on the host and was ignoring completely the Azure/Login action... This probably is related. Try giving the proper key-vault access to the managed identity assigned to the host where you're issuing the command from.

maonat avatar Oct 17 '24 11:10 maonat

I know this is an old post but... Where is this command being launched? I had the same issue on our self-hosted gh agents when updated to sops latest version and had to rollback to ensure this was working. In my case, the OS was authenticating to the keyvault using the managed identity on the host and was ignoring completely the Azure/Login action... This probably is related. Try giving the proper key-vault access to the managed identity assigned to the host where you're issuing the command from.

I launch it from my laptop. I use simple user credentials that being generated trough az login command. With this credentials I can access all secrets and keys in vault. Also as I said I even tried to use service principle with more broader permission, and it didnt work either.

andrey-gava avatar Oct 23 '24 19:10 andrey-gava

This error mentioned by other users: https://github.com/getsops/sops/issues/1415

andrey-gava avatar Oct 23 '24 19:10 andrey-gava

I know this is an old post but... Where is this command being launched? I had the same issue on our self-hosted gh agents when updated to sops latest version and had to rollback to ensure this was working. In my case, the OS was authenticating to the keyvault using the managed identity on the host and was ignoring completely the Azure/Login action... This probably is related. Try giving the proper key-vault access to the managed identity assigned to the host where you're issuing the command from.

I launch it from my laptop. I use simple user credentials that being generated trough az login command. With this credentials I can access all secrets and keys in vault. Also as I said I even tried to use service principle with more broader permission, and it didnt work either.

And what are the permissions? Are you assigning access policies ok the keyvault or using RBAC? (Or because it's either one or the other. If you have already an access policy in place in that keyvault, you won't be able to use RBAC.)

maonat avatar Oct 23 '24 20:10 maonat

And what are the permissions? Are you assigning access policies ok the keyvault or using RBAC? (Or because it's either one or the other. If you have already an access policy in place in that keyvault, you won't be able to use RBAC.)

We use RBAC access. My user role assignments on KeyVault and subscription level: [Key Vault Crypto Officer] [Key Vault Crypto User] [Contributor] [Reader]

Flow for user: Execute az login. Go trough multi-factor authentication. Try to decrypt file with 3.7.3 (OK). Try to decrypt file with 3.9.1 (ERROR)

| RESPONSE 403: 403 Forbidden
      | ERROR CODE: Forbidden
      | --------------------------------------------------------------------------------
      | {
      |   "error": {
      |     "code": "Forbidden",
      |     "message": "Caller is not authorized to perform action
      | on resource.\r\nIf role assignments, deny assignments or
      | role definitions were changed recently, please observe
      | propagation time.

Export mentioned here environments: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#hdr-User_with_username_and_password-EnvironmentCredential

export AZURE_TENANT_ID="******"
export AZURE_CLIENT_ID="******"
export AZURE_USERNAME="******"
export AZURE_PASSWORD="******"

Try to decrypt file with 3.7.3 (ERROR). Try to decrypt file with 3.9.1 (ERROR)

| {"error":"interaction_required","error_description":"AADSTS50076:
      | Due to a configuration change made by your administrator, or
      | because you moved to a new location, you must use
      | multi-factor authentication to access *****

Flow for service principle export AZURE_CONFIG_DIR="PATH_TO_NONE_DEFAULT_DIR" Execute az login --service-principal -u *** -p *** --tenant *** Try to decrypt file with 3.7.3 (OK). Try to decrypt file with 3.9.1 (ERROR)

| RESPONSE 403: 403 Forbidden
      | ERROR CODE: Forbidden
      | --------------------------------------------------------------------------------
      | {
      |   "error": {
      |     "code": "Forbidden",
      |     "message": "Caller is not authorized to perform action
      | on resource.\r\nIf role assignments, deny assignments or
      | role definitions were changed recently, please observe
      | propagation time.

Export mentioned here environments: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#hdr-Service_principal_with_client_secret-EnvironmentCredential

AZURE_TENANT_ID="******"
AZURE_CLIENT_ID="******"
AZURE_CLIENT_SECRET="******"

Try to decrypt file with 3.7.3 (OK). Try to decrypt file with 3.9.1 (OK)

So in my opinion looks like newer sops version cant get user token with Azure CLI credentials

az --version
azure-cli                         2.59.0 *

core                              2.59.0 *
telemetry                          1.1.0

Extensions:
account                            0.2.5
aks-preview                      0.5.173
desktopvirtualization               1.0.0
storage-preview                    0.8.4

Dependencies:
msal                              1.27.0
azure-mgmt-resource             23.1.0b2

Python location '/opt/az/bin/python3'
Extensions directory '/home/user/.azure/cliextensions'

Python (Linux) 3.11.8 (main, Mar 27 2024, 04:03:04) [GCC 11.4.0]

andrey-gava avatar Oct 24 '24 09:10 andrey-gava

Is there any update for this issue?

I know that using environment for AZURE_CLIENT_ID, AZURE_CLIENT_SECRET can be a workaround, but we are trying to federated credential instead of username/password in pipeline, so it does not works for us.

We are stuck on version 3.7.3 now with this problem.

xinfli avatar Dec 10 '24 14:12 xinfli

I faced this problem, located the issue, and found a solution. TLDR: AZURE_TOKEN_CREDENTIALS="dev"

Here's the longer story.

  1. Setup:
    • az version 2.77.0. sops version 3.11.0. Both latest, as of Sep 30th 2025.
    • I'm running sops encrypt --azure-kv … on a key vault that I definitely have access to.
    • My az CLI is logged in as a regular user. az account show shows a user with an email address, et al.
    • I was running all this from a VM that happens to have a Managed Identity. This was the root of the issue.
  2. I found these docs: https://github.com/Azure/azure-sdk-for-go/blob/main/sdk/azidentity/TROUBLESHOOTING.md#troubleshoot-defaultazurecredential-authentication-issues They suggested enabling logs, with AZURE_SDK_GO_LOGGING=all
  3. From sops with more logs, I could tell it was trying to use the managed identity from the VM, who does not have access to that key vault.
  4. Eventually, I found https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#hdr-Selecting_credentials-DefaultAzureCredential There, I learned to use either AZURE_TOKEN_CREDENTIALS="dev" or AZURE_TOKEN_CREDENTIALS="AzureCLICredential" to force sops to use the credentials from the az CLI.

I'll replicate those docs here (they were painfully difficult to find):

Set environment variable AZURE_TOKEN_CREDENTIALS to select a subset of the credential chain described above. DefaultAzureCredential will try only the specified credential(s), but its other behavior remains the same. Valid values for AZURE_TOKEN_CREDENTIALS are the name of any single type in the above chain, for example "EnvironmentCredential" or "AzureCLICredential", and these special values:

"dev": try AzureCLICredential and AzureDeveloperCLICredential, in that order "prod": try EnvironmentCredential, WorkloadIdentityCredential, and ManagedIdentityCredential, in that order

See also: https://learn.microsoft.com/en-gb/azure/developer/go/sdk/authentication/credential-chains

jpbochi avatar Sep 30 '25 21:09 jpbochi