Token validation fails when running multiple instances of IdentityServer4

[saisworld]

Issue

My identityserver4 (version 4.1.1) is deployed into AKS with replicas to 1. But when I set the replica count to more than 1, I start to see problems.

Sometimes token validation works sometimes it doesn't. After much closer look, I found that when the validatetoken request goes to the instance other than the one that created the token. This causes it to fail with error Tokenvalidation failed. As per the documentation, if more than one instance of the identityserver is running then we have to make sure that the same signing credentials should be used across all the instances. I have used Azure keyvault Keys(EC-256) for signing credentials following this article. But I'm still facing the problem. I have also made sure that the discovery url is same even when there are multiple instances runnings as per this link.

I have also used Keyvault and Storage account to store the the DataprotectionAPI keys.

Please advise if there is a different configuration that I have missed out for the token validation to work across multiple instances.

sample Startup code for fetching the Keys from Azure keyVault -

var keyClient = new KeyClient(
        new Uri(""), // e.g. https://scottbrady91-test.vault.azure.net/
        new ClientSecretCredential(
            tenantId: "",
            clientId: "",
            clientSecret: ""));

    Response<KeyVaultKey> response = keyClient.GetKey(""); // e.g. IdentityServerSigningKeyEcc

    AsymmetricSecurityKey key;
    string algorithm;

    if (response.Value.KeyType == KeyType.Ec)
    {
        ECDsa ecDsa = response.Value.Key.ToECDsa();
        key = new ECDsaSecurityKey(ecDsa) {KeyId = response.Value.Properties.Version};
        
        // parse from curve
        if (response.Value.Key.CurveName == KeyCurveName.P256) algorithm = "ES256";
        else if (response.Value.Key.CurveName == KeyCurveName.P384) algorithm = "ES384";
        else if (response.Value.Key.CurveName == KeyCurveName.P521) algorithm = "ES521";
        else  throw new NotSupportedException();
    }
    else if (response.Value.KeyType == KeyType.Rsa)
    {
        RSA rsa = response.Value.Key.ToRSA();
        key = new RsaSecurityKey(rsa) {KeyId = response.Value.Properties.Version};

        // you define
        algorithm = "PS256";
    }
    else
    {
        throw new NotSupportedException();
    }
    
    services.AddIdentityServer()
        .AddSigningCredential(key, algorithm)
       .AddSecretValidator<OAuthSecretValidator>()   //Adds a custom OAuth Secret Validator into the chain to handle original validation method
       .AddProfileService<UserProfileService>()  //Adds the service that interacts with our custom UserStore
       .AddClientStore<CustomClientStore>() //Adds the custom client store
       .AddResourceStore<CustomResourceStore>();  //Adds a custom resource store for both Api and Identity resources

  services.AddTransient<IUserStore, CustomUserStore>();  //Register our custom user store