Exception - Microsoft.Bot.Builder.AI.Orchestrator.OrchestratorRecognizer: "Failed to find or load Model with path" Exception during initialization: bad allocation

[dawwa]

Describe the bug

After the bot web app running for a few days, started getting this exception. This consistently repros for web app services that have multiple instances, and we have to scale up and down to get rid of the bad nodes. Would you please help to understand what could be the possible reason? and how should we investigate?

Exception: Failed to find or load Model with path C:\home\site\wwwroot\generated\orchestratorModels\english

Call Stack: System.InvalidOperationException Microsoft.Bot.Builder.AI.Orchestrator.OrchestratorRecognizer.<InitializeModel>b__39_0 [external code] Microsoft.Bot.Builder.BotAdapter+<RunPipelineAsync>d__23.MoveNext Inner exception System.ApplicationException handled at Microsoft.Bot.Builder.AI.Orchestrator.OrchestratorRecognizer.<InitializeModel>b__39_0 Microsoft.BotFramework.Orchestrator.Orchestrator..ctor Microsoft.Bot.Builder.AI.Orchestrator.OrchestratorRecognizer.<InitializeModel>b__39_0

[MahmoudNawar]

I can confirm this happens in the centos stream. I don't find any reasonable way to find the source of this issue. the app is working fine and just start failing in loading the model " Failed to find or load Model with path /opt/chatbots/models/pretrained.20210205.microsoft.dte.00.06.unicoder_multilingual "