microsoft
Cannot run model generated with recipe on AMD NP with Foundry Local v0.8
Created from comment in #247
@natke I used olive to convert custom model to be used with npu, I was able to use this with foundry v6 but with upgrade to v8 (to be able to use amd strix halo npu), I am getting below error. I have no issues in running the models provided with npu support on foundry local catalogue, is there any more documentation on library versions compatibility besides: https://learn.microsoft.com/en-us/azure/ai-foundry/foundry-local/how-to/how-to-compile-hugging-face-models?view=foundry-classic https://github.com/microsoft/Foundry-Local/issues/65
olive auto-opt --model_name_or_path local_model
--output_path models/7B-ONNX-INT4-Olive-NPU-FoundryLocal --device npu
--provider VitisAIExecutionProvider --use_ort_genai
--precision int4 --log_level 3
--use_model_builder
Libraries
olive-ai @ git+https://github.com/microsoft/Olive.git@8b44cf49e03b8c4bf5d9f31ec3aa6a5f22d7677d onnx==1.18.0 onnx-ir==0.1.12 onnx-tool==0.9.0 onnxoptimizer @ file:///C:/Program%20Files/RyzenAI/1.6.1/onnxoptimizer-0.3.19-cp312-cp312-win_amd64.whl#sha256=5ae6edeea455859216802a939fe7b4477122014fc42dc51f738614d1298546eb onnxruntime-genai-directml-ryzenai @ file:///C:/Program%20Files/RyzenAI/1.6.1/onnxruntime_genai_directml_ryzenai-0.9.2-cp312-cp312-win_amd64.whl#sha256=496f23d9b47d7deaf8fcb58cbd17fab9753227b91b855c43c4908d4e2e44784a onnxruntime-vitisai @ file:///C:/Program%20Files/RyzenAI/1.6.1/onnxruntime_vitisai-1.23.2-cp312-cp312-win_amd64.whl#sha256=4b48265bd2754f05a9bcbdfc5c54ea185a832c73d3fe61df15fadc4d8023b6d8 onnxruntime_extensions==0.13.0 onnxscript==0.5.6 onnxsim @ file:///C:/Program%20Files/RyzenAI/1.6.1/onnxsim-0.4.36-cp312-cp312-win_amd64.whl#sha256=da202a54288196105d1371a5dc0a792f2e9d97065c7778508a4021229111cb37 Error
Interactive Chat. Enter /? or /help for help. Press Ctrl+C to cancel generation. Type /exit to leave the chat.
Interactive mode, please enter your prompt
Hi 🧠 Thinking [11:36:14 ERR] Connection id "0HNH8LSA3UHVR", Request id "0HNH8LSA3UHVR:00000001": An unhandled exception was thrown by the application. Exception: Error during chat The response ended prematurely. (ResponseEnded) [11:36:14 ERR] LogException Microsoft.AI.Foundry.Local.Common.FLException: Error during chat ---> System.Net.Http.HttpIOException: The response ended prematurely. (ResponseEnded) at System.Net.Http.HttpConnection.<FillAsync>d__82.MoveNext() + 0x265 --- End of stack trace from previous location --- at System.Net.Http.HttpConnection.ChunkedEncodingReadStream.<ReadAsyncCore>d__6.MoveNext() + 0x25b --- End of stack trace from previous location --- at System.ClientModel.Internal.ReadTimeoutStream.<ReadAsync>d__21.MoveNext() + 0x24d --- End of stack trace from previous location --- at System.Net.ServerSentEvents.SseParser
1.<FillLineBufferAsync>d__35.MoveNext() + 0x1ba --- End of stack trace from previous location --- at System.Net.ServerSentEvents.SseParser1.<EnumerateAsync>d__20.MoveNext() + 0x241 --- End of stack trace from previous location --- at OpenAI.AsyncSseUpdateCollection1.AsyncSseUpdateEnumerator1.<System-Collections-Generic-IAsyncEnumerator<U>-MoveNextAsync>d__12.MoveNext() + 0x1de --- End of stack trace from previous location --- at OpenAI.AsyncSseUpdateCollection1.<GetValuesFromPageAsync>d__8.MoveNext() + 0x261 --- End of stack trace from previous location --- at OpenAI.AsyncSseUpdateCollection1.<GetValuesFromPageAsync>d__8.MoveNext() + 0x3ba --- End of stack trace from previous location --- at System.ClientModel.AsyncCollectionResult1.<GetAsyncEnumerator>d__1.MoveNext() + 0x339 --- End of stack trace from previous location --- at System.ClientModel.AsyncCollectionResult1.<GetAsyncEnumerator>d__1.MoveNext() + 0x69f --- End of stack trace from previous location --- at System.ClientModel.AsyncCollectionResult1.<GetAsyncEnumerator>d__1.MoveNext() + 0x7d4 --- End of stack trace from previous location --- at Microsoft.AI.Foundry.Local.Commands.ModelRunCommand.<InteractiveNewRoundAsync>d__7.MoveNext() + 0x2a0 --- End of stack trace from previous location --- at Microsoft.AI.Foundry.Local.Commands.ModelRunCommand.<InteractiveNewRoundAsync>d__7.MoveNext() + 0x618 --- End of inner exception stack trace --- at Microsoft.AI.Foundry.Local.Commands.ModelRunCommand.HandleExceptionDuringChat(Exception) + 0xee at Microsoft.AI.Foundry.Local.Commands.ModelRunCommand.<InteractiveNewRoundAsync>d__7.MoveNext() + 0x804 --- End of stack trace from previous location --- at Microsoft.AI.Foundry.Local.Commands.ModelRunCommand.<RunInteractiveNewRoundAsync>d__3.MoveNext() + 0x5c --- End of stack trace from previous location --- at Microsoft.AI.Foundry.Local.Commands.ModelRunCommand.<<Create>b__1_0>d.MoveNext() + 0x172e --- End of stack trace from previous location --- at Microsoft.AI.Foundry.Local.Common.CommandActionFactory.<>c__DisplayClass0_01.<<Create>b__0>d.MoveNext() + 0x1e7 --- End of stack trace from previous location --- at System.CommandLine.NamingConventionBinder.CommandHandler.<GetExitCodeAsync>d__66.MoveNext() + 0x50 --- End of stack trace from previous location --- at System.CommandLine.NamingConventionBinder.ModelBindingCommandHandler.<InvokeAsync>d__11.MoveNext() + 0x61 --- End of stack trace from previous location --- at System.CommandLine.Invocation.InvocationPipeline.<InvokeAsync>d__0.MoveNext() + 0x1cd --- End of stack trace from previous location --- at Microsoft.AI.Foundry.Local.Program.<Main>d__1.MoveNext() + 0x4e4 Please check the service status with 'foundry service status'.
sunilkumar-rakuten Can you please set verbose logging on
foundry service set --default-log-level Debug
And then check the .foundry/logs folder and send the logs from the failed call