[BUG] Unable to use PEM certificate in Azure.Identity 1.4.0 and beyond

[jarz]

Library name and version

Azure.Identity 1.4.0+

Describe the bug

My team has been utilizing the service principal certificate (spncert.pem) in Azure DevOps to authenticate a custom runner against an Azure service (Digital Twins). The runner authenticates successfully with Azure.Identity 1.3.0.

Upgrading the Azure.Identity NuGet package to 1.4.0 and later breaks this authentication with the following inner exception: System.ArgumentException: The provided key does not match the public key for this certificate. (Parameter 'privateKey')

A DefaultAzureCredential is used with the following environment variables set:

• AZURE_CLIENT_ID
• AZURE_TENANT_ID
• AZURE_CLIENT_CERTIFICATE_PATH

Expected behavior

Successful authentication using the service principal certificate given environment variables correctly set.

Actual behavior

Relevant portion of stack trace:

 ---> Azure.Identity.CredentialUnavailableException: Could not load certificate file
 ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.ArgumentException: The provided key does not match the public key for this certificate. (Parameter 'privateKey')
   at System.Security.Cryptography.X509Certificates.RSACertificateExtensions.CopyWithPrivateKey(X509Certificate2 certificate, RSA privateKey)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at Azure.Core.PemReader.CreateRsaCertificate(Byte[] cer, Byte[] key, X509KeyStorageFlags keyStorageFlags)
   at Azure.Core.PemReader.LoadCertificate(ReadOnlySpan`1 data, Byte[] cer, KeyType keyType, Boolean allowCertificateOnly, X509KeyStorageFlags keyStorageFlags)
   at Azure.Identity.X509Certificate2FromFileProvider.LoadCertificateFromPemFileAsync(Boolean async, String clientCertificatePath, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Azure.Identity.X509Certificate2FromFileProvider.LoadCertificateFromPemFileAsync(Boolean async, String clientCertificatePath, CancellationToken cancellationToken)
   at Azure.Identity.MsalConfidentialClient.CreateClientAsync(Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.MsalClientBase`1.GetClientAsync(Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.MsalConfidentialClient.AcquireTokenForClientCoreAsync(String[] scopes, String tenantId, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.MsalConfidentialClient.AcquireTokenForClientAsync(String[] scopes, String tenantId, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.ClientCertificateCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage)
   at Azure.Identity.ClientCertificateCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.EnvironmentCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage)
   at Azure.Identity.EnvironmentCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.EnvironmentCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)

Reproduction Steps

1. Create an Azure Resource Manager service connection in Azure DevOps.
2. In an Azure pipeline, use the service connection to authenticate in an AzureCLI@2 task.
   1. Ensure the azureSubscription input is set to the service connection name.
   2. addSpnToEnvironment should be true.
   3. We use an inlineScript to set the AZURE_CLIENT_ID, AZURE_TENANT_ID, and AZURE_CLIENT_CERTIFICATE_PATH environment variables.
3. In the next pipeline task, run a compiled EXE that relies on DefaultAzureCredential for authentication. The EXE will work for Azure.Identity 1.3.0, but not for later versions.

Environment

Azure DevOps - Windows agent

[m-redding]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[christothes]

Hi @jarz - Are you able to reproduce this outside of the Azure pipeline using just the DefaultAzureCredential or ClientCertificateCredential and any valid PEM?

[jarz]

I'm not able to reproduce this outside of the pipeline. I don't have permissions to add a new certificate to the application registration, so that limits my options for validation.

Locally, I've used a self-generated certificate (using OpenSSL) with otherwise correct AZURE_CLIENT_ID and AZURE_TENANT_ID values. That results in an expected 401 from AAD, stating the key was not found using both Azure.Identity 1.3.0 and 1.6.1. That appears to be further along in the auth flow than the stack trace I posted above.

Overall, it appears Azure.Identity 1.4.0+ is unable to parse an otherwise acceptable PEM file. Again, I've tried to find details on the content of spncert.pem, but have found nothing.

[christothes]

Hi @jarz Could you provide any details on how you get the path to the certificate in question? I should be able to reproduce and investigate the root cause.

A couple other questions

• If you use the $env:servicePrincipalId, $env:servicePrincipalKey and $env:tenantId values to set the AZURE_CLIENT_ID, AZURE_TENANT_ID, and AZURE_CLIENT_SECRET variables, does the EnvironmentCredential work for you?
• Does this reproduce on a Linux agent?

[ghost]

Hi, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[jarz]

Hey @christothes, the certificate has a known location on Windows agents: D:\a\_temp\spnCert.pem. The certificate is there when the AzureCLI@2 task has the addSpnToEnvironment set to true (see here).

This particular task is used in a couple internal repositories. One of our service connections relies on the client secret and the EnvironmentCredential is successfully used instead.

Our team does not use Linux agents for our pipelines, so I can't say if it's an issue on that platform. The path would certainly be different.

If needed, ping me on Teams and I can give you the task definition.

[christothes]

Looks like this is an instance of https://github.com/Azure/azure-sdk-for-net/issues/19043 where the PEM has a certificate chain and we are not properly selecting the leaf cert.