Help to clarify the ARM token cache for managed identity login

[northtyphoon]

Related command az login --identity any az command

Describe the bug I run az commands in Azure VM with managed identity enabled. I log into cli using "az login --identity".

After login, I noticed all az commands always tried to connect "http://169.254.169.254/metadata/identity/oauth2/token" to get ARM token when the corresponding code tried to query ARM resource. I'd expect CLI could cache the ARM token and tried to reuse it if possible. I shared one debug output as below. As you can also see, after the token request, it mentioned "MSI: token is found in cache". If the token could be found, why the code didn't check it at first place?

cli.azure.cli.core.commands.client_factory: Getting management service client client_type=ComputeManagementClient urllib3.connectionpool: Starting new HTTP connection (1): 169.254.169.254:80 urllib3.connectionpool: http://169.254.169.254:80 "GET /metadata/identity/oauth2/token?resource=https%3A%2F%2Fmanagement.core.windows.net%2F&api-version=2018-02-01 HTTP/1.1" 200 1739 msrestazure.azure_active_directory: MSI: Retrieving a token from http://169.254.169.254/metadata/identity/oauth2/token, with payload {'resource': 'https://management.core.windows.net/', 'api-version': '2018-02-01'} msrestazure.azure_active_directory: MSI: Token retrieved cli.azure.cli.core.auth.adal_authentication: MSIAuthenticationWrapper.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={} msrestazure.azure_active_directory: MSI: token is found in cache. cli.azure.cli.core.auth.adal_authentication: Normalize expires_on: '1663545077' -> 1663545077

To Reproduce

1. set up an Azure VM with manage identity enabled
2. install the latest azure-cli
3. az login --identity
4. run any az command to get the resources a few times like

az vmss list --debug az storage list --debug az acr list --debug

Expected behavior ARM token is cached and reused if not expired.

Environment summary CLI: 2.40.0 Azure VM with manage identity enabled

[jiasli]

msrestazure.azure_active_directory: MSI: token is found in cache.

This line is actually from https://github.com/Azure/msrestazure-for-python/blob/8849f398b6ebd4607de63c2f5d1318f44ec1d822/msrestazure/azure_active_directory.py#L644

    def get_token(self, resource):
        ...
                _LOGGER.debug("MSI: token is found in cache.")
                return token_entry

_ImdsTokenProvider.get_token was called by MSIAuthentication.set_token.

https://github.com/Azure/msrestazure-for-python/blob/8849f398b6ebd4607de63c2f5d1318f44ec1d822/msrestazure/azure_active_directory.py#L600

    def set_token(self):
        ...
            token_entry = self._vm_msi.get_token(self.resource)

MSIAuthentication.set_token was first called by

https://github.com/Azure/msrestazure-for-python/blob/8849f398b6ebd4607de63c2f5d1318f44ec1d822/msrestazure/azure_active_directory.py#L592

    def __init__(self, port=50342, **kwargs):
        ...
        self.set_token()

After MSIAuthentication is instantiated, Azure CLI also calls MSIAuthentication.set_token at https://github.com/Azure/azure-cli/blob/0fb75afa1e75f8a2419514c5fcf19533138f7185/src/azure-cli-core/azure/cli/core/auth/adal_authentication.py#L46

leading to the token is found in cache message.

We can easily see that this cache is only an in-memory cache:

https://github.com/Azure/msrestazure-for-python/blob/8849f398b6ebd4607de63c2f5d1318f44ec1d822/msrestazure/azure_active_directory.py#L632

    def __init__(self, msi_conf=None, timeout=None):
        ...
        self.cache = {}

Each Azure CLI command is run in a separate process, so each Azure CLI command can't share the same in-memory cache. The only way to share cache between processes is to persist the cache to disk.

http://169.254.169.254/metadata/identity/oauth2/token is a special endpoint which is very quick. No outbound internet connection is made.

Also, Managed identity endpoint http://169.254.169.254/metadata/identity/oauth2/token has its own caching mechanism, so developing another cache to persist the access token on disk is neither necessary nor secure.

[northtyphoon]

@jiasli I can understand different commands won't be able to use the in-memory cache. But I also see in the same command, if it calls ARM API several times, it will also call http://169.254.169.254/metadata/identity/oauth2/token each time. How can the command author to utilize the in-memory cache? Also if I use service principal or normal aad account to login, how does the token cache work?

[jiasli]

But I also see in the same command, if it calls ARM API several times, it will also call http://169.254.169.254/metadata/identity/oauth2/token each time.

Could you let us know which command you are calling? I suspect the command is creating multiple client factories thus multiple MSIAuthenticationWrappers.

Also if I use service principal or normal aad account to login, how does the token cache work?

In short, we are using msal-extensions's persisted token cache. We can discuss internally if you like.

[northtyphoon]

the command I called is az acr repository show-tags