PowerPlatform-DataverseServiceClient icon indicating copy to clipboard operation
PowerPlatform-DataverseServiceClient copied to clipboard
verified publisher icon microsoft

Failed to connect to Dataverse One or more errors occurred in azure function

Open v-leonhou opened this issue 1 year ago • 10 comments

Failed to connect to Dataverse One or more errors occurred. (Unable to cast object of type 'generatedProxy_14' to type 'Microsoft.PowerPlatform.Dataverse.Client.IOrganizationServiceAsync'.) Unable to cast object of type 'generatedProxy_14' to type 'Microsoft.PowerPlatform.Dataverse.Client.IOrganizationServiceAsyn

dataverse client sdk version:1.0.26 and 1.0.39 ( both has the issue)

azure function runtime version:: 3.23.0.0

App Service Plan： EP1:2(price level:EP1, instance count:2 )

v-leonhou avatar Dec 03 '24 13:12 v-leonhou

This error has appeared serval times and seems to be related to Azure Functions Scale Out mechanics and the use of .Clone.

Is this error happening consistently? or intermittently? also are you using .clone in your code?

Please see: #295 and #259

MattB-msft avatar Dec 05 '24 19:12 MattB-msft

hello, this is our code, the error happened normally intermittently , but last week the error suddenly happended consistently and it continued almost three days

public class CommonHelper {

    private static Lazy<ServiceClient> _LazyServiceClient = new Lazy<ServiceClient>(() => CreateOrganizationService("ConnectionString"), true);


    private static Lazy<ServiceClient> _LazyServiceClient_AddMaterialInfo = new Lazy<ServiceClient>(() => CreateOrganizationService("ConnectionString_AddMaterialInfo"), true);

    private static Lazy<ServiceClient> _LazyServiceClient_ServiceComplete = new Lazy<ServiceClient>(() => CreateOrganizationService("ConnectionString_ServiceComplete"), true);

    private static Lazy<ServiceClient> _LazyServiceClient_WarrantyQuery = new Lazy<ServiceClient>(() => CreateOrganizationService("ConnectionString_WarrantyQuery"), true);

    /// <returns></returns>
    public static ServiceClient GetOrganizationService(ILogger log = null, string connectionKey = "ConnectionString")
    {
        ServiceClient organizationService = null;
        if (connectionKey == "ConnectionString_AddMaterialInfo")
            organizationService = _LazyServiceClient_AddMaterialInfo.Value.Clone(log);
        else if (connectionKey == "ConnectionString_ServiceComplete")
            organizationService = _LazyServiceClient_ServiceComplete.Value.Clone(log);
        else if (connectionKey == "ConnectionString_WarrantyQuery")
            organizationService = _LazyServiceClient_WarrantyQuery.Value.Clone(log);
        else
            organizationService = _LazyServiceClient.Value.Clone(log);
        log?.LogInformation($"[{connectionKey}] OrganizationService IsReady={organizationService.IsReady}");
        if (!organizationService.IsReady)
            throw new Exception($"[{connectionKey}] OrganizationService IsReady is false");
        string isEnableAffinityCookie = Environment.GetEnvironmentVariable("ConnectionString_IsEnableAffinityCookie");
        if (organizationService.EnableAffinityCookie && !string.IsNullOrWhiteSpace(isEnableAffinityCookie) && new string[] { "false", "0" }.Contains(isEnableAffinityCookie))
            organizationService.EnableAffinityCookie = false;
        log?.LogInformation($"[{connectionKey}] OrganizationService EnableAffinityCookie={organizationService.EnableAffinityCookie}");
        return organizationService;
    }

    /// <param name="connectionKey"></param>
    /// <param name="log"></param>
    /// <returns></returns>
    public static ServiceClient CreateOrganizationService(string connectionKey, ILogger log = null)
    {
        string connectionString = Environment.GetEnvironmentVariable(connectionKey).Trim();
        if (string.IsNullOrWhiteSpace(connectionString))
            throw new Exception($"Environment Variable [{connectionKey}] is null, please copy this error and contact ISP support");
        int retryCount = 0;
        ServiceClient organizationService = null;
        string failmsg = string.Empty;
        while (retryCount < 5 && organizationService == null)
        {
            retryCount++;
            try
            {
                organizationService = new ServiceClient(connectionString, log);
            }
            catch (Exception ex)
            {
                string clientId = ExtractClientIdFromConnectionString(connectionString);
                log?.LogError($"ClientID: {clientId}, Error: {ex}");
                log?.LogError(ex, $"第{retryCount}次GetOrganizationService失败");
                failmsg = ex.Message;
            }
        }
        if (organizationService == null)
            //throw new Exception($"[{connectionString}] create ServiceClient 失败：{failmsg}");
            throw new Exception($"Create ServiceClient failed: {failmsg}, please copy this error and contact ISP support");
        return organizationService;
    }

}`

v-leonhou avatar Dec 06 '24 07:12 v-leonhou

Clone has Known issues with Functions, Please move over to using Async Execution in your functions, using a single instance of the connection (or set of connections)

for example, CreateAsync , RetrieveMutlipleAsync, etc,

thanks

MattB-msft avatar Dec 10 '24 20:12 MattB-msft

We also started getting this same error in our production environment 3 days ago and we are unable to reproduce the error in our pre-production environment.

It happens intermittently. After restarting the function app the error disappears.

The error first appeared when Microsoft.PowerPlatform.Dataverse.Client was on version 1.0.9, there after we upgraded to 1.1.32 and this morning the error occurred again.

Function app runtime version : ~4 .net version 6.0

This is our code.

services.AddSingleton<IOrganizationServiceAsync, ServiceClient>(provider =>
{
        return GetService(d365Settings);
});

public static ServiceClient GetService(D365Settings d365Settings)
{
	var client = new ServiceClient(new Uri(d365Settings.OrganizationUri), d365Settings.ClientId, d365Settings.ClientSecret, false);

	if (client.IsReady)
		return client;

	throw new ArgumentException($"Unable to connect to CE, Error: {client.LastError}", client.LastException);
}

Any help would be much appropriated.

Sakkie avatar Dec 13 '24 07:12 Sakkie

@Sakkie Thanks for your code sample here.

In your code where you inject the Singelton into the invoking class, are you cloning the returned client there?

MattB-msft avatar Jan 10 '25 00:01 MattB-msft

@MattB-msft Thanks for responding to my comment.

No, we aren't using .clone at all.

The system was working fine up to the 6th of Dec 2024. We deployed some optimization on another part of the system and then we started seeing this error a few times a day.

After restarting the function app, the error stops for a few hours.

Sakkie avatar Jan 10 '25 07:01 Sakkie

Then this is new, and your seeing this error: Unable to cast object of type 'generatedProxy_14' to type 'Microsoft.PowerPlatform.Dataverse.Client.IOrganizationServiceAsync'.) Unable to cast object of type 'generatedProxy_14' to type 'Microsoft.PowerPlatform.Dataverse.Client.IOrganizationServiceAsyn

Are you running a Linux or Windows host for the function? Also can you provide a stack trace or logs to help us understand at what point in your code where this is surfacing?

MattB-msft avatar Jan 11 '25 02:01 MattB-msft

We are running a Windows host. Solution is .net 6 (I'm in the process to upgrade it to .net 8)

Below are the stack trace for the above error

Microsoft.Azure.WebJobs.Host.FunctionInvocationException:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:352)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsync>d__18.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:108)
Inner exception Microsoft.PowerPlatform.Dataverse.Client.Utils.DataverseConnectionException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
   at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient.CreateServiceConnection (Microsoft.PowerPlatform.Dataverse.Client, Version=1.1.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient..ctor (Microsoft.PowerPlatform.Dataverse.Client, Version=1.1.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at CompanyName.Integration.SystemName.Infrastructure.Italy.ServiceRegistration.GetService (CompanyName.Integration.SystemName.Infrastructure.Italy, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\current\src\acapp\SystemName\Infrastructure\CompanyName.Integration.SystemName.Infrastructure.Italy\ServiceRegistration.cs:42)
   at CompanyName.Integration.SystemName.Infrastructure.Italy.ServiceRegistration+<>c__DisplayClass1_0.<AddNoveoD365ItalyInfrastructure>b__1 (CompanyName.Integration.SystemName.Infrastructure.Italy, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\current\src\acapp\SystemName\Infrastructure\CompanyName.Integration.SystemName.Infrastructure.Italy\ServiceRegistration.cs:36)
   at DryIoc.Microsoft.DependencyInjection.DryIocAdapter+<>c__DisplayClass3_0.<RegisterDescriptor>b__0 (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.636.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/DryIocAdapter.cs:162)
   at DryIoc.Registrator+<>c__DisplayClass27_0.<RegisterDelegate>b__0 (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.636.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:4550)
   at System.Linq.Expressions.Interpreter.FuncCallInstruction`3.Run (System.Linq.Expressions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Linq.Expressions.Interpreter.Interpreter.Run (System.Linq.Expressions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Linq.Expressions.Interpreter.LightLambda.Run (System.Linq.Expressions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Dynamic.Utils.DelegateHelpers.FuncThunk1 (System.Linq.Expressions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at DryIoc.Factory+<>c__DisplayClass26_0.<ApplyReuse>b__2 (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.636.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6605)
   at DryIoc.Scope.TryGetOrAdd (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.636.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:7850)
   at DryIoc.Scope.GetOrAdd (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.636.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:7835)
   at DryIoc.Factory.ApplyReuse (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.636.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6604)
   at DryIoc.Factory.GetExpressionOrDefault (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.636.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6564)
   at DryIoc.Factory.GetDelegateOrDefault (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.636.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6634)
   at DryIoc.DelegateFactory.GetDelegateOrDefault (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.636.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:7741)
   at DryIoc.Container.DryIoc.IResolver.Resolve (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.636.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:290)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Linq.Expressions.Interpreter.ExceptionHelpers.UnwrapAndRethrow (System.Linq.Expressions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Linq.Expressions.Interpreter.MethodInfoCallInstruction.Run (System.Linq.Expressions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Linq.Expressions.Interpreter.Interpreter.Run (System.Linq.Expressions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Linq.Expressions.Interpreter.LightLambda.Run (System.Linq.Expressions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Dynamic.Utils.DelegateHelpers.FuncThunk (System.Linq.Expressions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at DryIoc.Scope.TryGetOrAdd (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.636.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:7850)
   at DryIoc.Scope.GetOrAdd (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.636.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:7835)
   at DryIoc.CurrentScopeReuse.GetScopedOrSingleton (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.636.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:8167)
   at System.Linq.Expressions.Interpreter.FuncCallInstruction`5.Run (System.Linq.Expressions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Linq.Expressions.Interpreter.Interpreter.Run (System.Linq.Expressions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Linq.Expressions.Interpreter.LightLambda.Run (System.Linq.Expressions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Dynamic.Utils.DelegateHelpers.FuncThunk (System.Linq.Expressions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at DryIoc.Scope.TryGetOrAdd (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.636.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:7850)
   at DryIoc.Scope.GetOrAdd (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.636.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:7835)
   at DryIoc.CurrentScopeReuse.GetScopedOrSingleton (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.636.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:8167)
   at System.Linq.Expressions.Interpreter.FuncCallInstruction`5.Run (System.Linq.Expressions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Linq.Expressions.Interpreter.Interpreter.Run (System.Linq.Expressions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Linq.Expressions.Interpreter.LightLambda.Run (System.Linq.Expressions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Dynamic.Utils.DelegateHelpers.FuncThunk1 (System.Linq.Expressions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at DryIoc.Container.ResolveAndCacheDefaultFactoryDelegate (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.636.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:224)
   at DryIoc.Container.DryIoc.IResolver.Resolve (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.636.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:195)
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.ScopedServiceProvider.GetService (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.636.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/ScopedServiceProvider.cs:25)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService (Microsoft.Extensions.DependencyInjection.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService (Microsoft.Extensions.DependencyInjection.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at MediatR.Wrappers.RequestHandlerWrapperImpl`2+<>c__DisplayClass1_0.<Handle>g__Handler|0 (MediatR, Version=12.0.0.0, Culture=neutral, PublicKeyToken=bb9a41a5e8aaa7e2)
   at CompanyName.Integration.SystemName.Application.Behaviours.ValidationBehavior`2+<Handle>d__2.MoveNext (CompanyName.Integration.SystemName.Application, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\current\src\acapp\SystemName\Core\CompanyName.Integration.SystemName.Application\Behaviours\ValidationBehavior.cs:27)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at CompanyName.Integration.SystemName.FunctionApp.Functions.v1.AppointmentsFunction+<GetQueue>d__11.MoveNext (CompanyName.Integration.SystemName.FunctionApp, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\current\src\acapp\SystemName\FunctionApp\CompanyName.Integration.SystemName.FunctionApp\Functions\v1\AppointmentsFunction.cs:221)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2+<InvokeAsync>d__2.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\VoidTaskMethodInvoker.cs:20)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+<InvokeAsync>d__10.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:53)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<InvokeWithTimeoutAsync>d__33.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:581)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__32.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:527)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:306)
Inner exception System.InvalidCastException handled at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient.CreateServiceConnection:
   at System.Runtime.CompilerServices.CastHelpers.ChkCastAny (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Reflection.DispatchProxy.Create (System.Reflection.DispatchProxy, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.ServiceModel.Channels.ServiceChannelProxy.CreateProxy (System.ServiceModel.Primitives, Version=6.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.ServiceModel.Channels.ServiceChannelFactory.CreateProxy (System.ServiceModel.Primitives, Version=6.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.ServiceModel.Channels.ServiceChannelFactory.CreateChannel (System.ServiceModel.Primitives, Version=6.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.ServiceModel.ChannelFactory`1.CreateChannel (System.ServiceModel.Primitives, Version=6.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.ServiceModel.ChannelFactory`1.CreateChannel (System.ServiceModel.Primitives, Version=6.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.ServiceModel.ClientBase`1.CreateChannel (System.ServiceModel.Primitives, Version=6.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.ServiceModel.ClientBase`1.CreateChannelInternal (System.ServiceModel.Primitives, Version=6.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.ServiceModel.ClientBase`1.get_Channel (System.ServiceModel.Primitives, Version=6.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.ServiceModel.ClientBase`1.get_InnerChannel (System.ServiceModel.Primitives, Version=6.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService+<ConnectAndInitServiceAsync>d__204.MoveNext (Microsoft.PowerPlatform.Dataverse.Client, Version=1.1.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService+<DoDirectLoginAsync>d__185.MoveNext (Microsoft.PowerPlatform.Dataverse.Client, Version=1.1.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService+<InitServiceAsync>d__184.MoveNext (Microsoft.PowerPlatform.Dataverse.Client, Version=1.1.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.GetCachedService (Microsoft.PowerPlatform.Dataverse.Client, Version=1.1.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.IntilizeService (Microsoft.PowerPlatform.Dataverse.Client, Version=1.1.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient.CreateServiceConnection (Microsoft.PowerPlatform.Dataverse.Client, Version=1.1.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)

One thing I did not mention before, we get 2 errors randomly at the same times during the day while the system is being used. The above one and another.

Both are System.InvalidCastException

The seconds one doesnt make sense as there are only one ShopModel in the solution. So to me it feels like something goes wrong with the DI when new hosts instance are initialized.

The message is

[A]CompanyName.Integration.SystemName.Domain.Models.DataStorageModels.ShopModel cannot be cast to [B]CompanyName.Integration.SystemName.Domain.Models.DataStorageModels.ShopModel. Type A originates from 'CompanyName.Integration.SystemName.Domain, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' in the context 'Default' at location 'C:\home\site\wwwroot\bin\CompanyName.Integration.SystemName.Domain.dll'. Type B originates from 'CompanyName.Integration.SystemName.Domain, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' in the context 'Default' at location 'C:\home\site\wwwroot\bin\CompanyName.Integration.SystemName.Domain.dll'.

Stack trace is

System.InvalidCastException:
   at System.Runtime.CompilerServices.CastHelpers.ChkCastAny (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Extensions.Caching.Memory.CacheExtensions+<GetOrCreateAsync>d__9`1.MoveNext (Microsoft.Extensions.Caching.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at CompanyName.Integration.SystemName.Infrastructure.DataStorage.Services.DataStorageService+<GetShopAsync>d__12.MoveNext (CompanyName.Integration.SystemName.Infrastructure.DataStorage, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\current\src\acapp\SystemName\Infrastructure\CompanyName.Integration.SystemName.Infrastructure.DataStorage\Services\DataStorageService.cs:53)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at CompanyName.Integration.SystemName.Application.Features.Appointments.Commands.QueueAppointment.CreateQueueAppointmentCommandHandler+<Handle>d__6.MoveNext (CompanyName.Integration.SystemName.Application, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\current\src\acapp\SystemName\Core\CompanyName.Integration.SystemName.Application\Features\Appointments\Commands\QueueAppointment\QueueAppointmentCommand.cs:52)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at CompanyName.Integration.SystemName.Application.Behaviours.ValidationBehavior`2+<Handle>d__2.MoveNext (CompanyName.Integration.SystemName.Application, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\current\src\acapp\SystemName\Core\CompanyName.Integration.SystemName.Application\Behaviours\ValidationBehavior.cs:27)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at CompanyName.Integration.SystemName.FunctionApp.Functions.v1.AppointmentsFunction+<QueueAppointment>d__13.MoveNext (CompanyName.Integration.SystemName.FunctionApp, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\current\src\acapp\SystemName\FunctionApp\CompanyName.Integration.SystemName.FunctionApp\Functions\v1\AppointmentsFunction.cs:250)

The system would function fine for a few hours, then suddenly I would see a few hundred or thousand of the above Exceptions. After restartign the system all returns back to normal for a few hours again.

Sakkie avatar Jan 13 '25 10:01 Sakkie 

public static ServiceClient GetService(D365Settings d365Settings)
{
	var client = new ServiceClient(new Uri(d365Settings.OrganizationUri), d365Settings.ClientId, d365Settings.ClientSecret, false);

	if (client.IsReady)
		return client;

	throw new ArgumentException($"Unable to connect to CE, Error: {client.LastError}", client.LastException);
}

Any help would be much appropriated.

Are you recreating it if something goes wrong? - In my experience a lot of time the ServiceClient fails and get in an invalid state where its unable to return to a functionel state.

We had similar issues after we basicly wrapped every call in code similar to below we removed almost all issues with connecting to dataverse.

Try { CallDataverse(call){ } Catch (Exception) { ReCreateServiceClient() CallDataverse(call) }

FarmerenHE avatar May 02 '25 21:05 FarmerenHE

Any update on this issue? We seem to be encountering it on 1.0.39 in a .NET 6.0 Function App.

We're not using .Clone()

Tried updating to a later version, but that's causing other issues in our project at the moment, so will have a look at fixing that, but as this is still open I'm assuming it may not be fixed in any version?

Seems to occur after a few minutes of running successfully, so guessing it could be a scaling out issue. Once it has started failing, all DataverseClient instances fail until the Function App is restarted.

richgp avatar Jul 04 '25 08:07 richgp