Sidekick icon indicating copy to clipboard operation
Sidekick copied to clipboard
Published 2 months ago • verified publisher icon johnbean393

BUG: Sidekick not seeing local models

Open spacemonkey opened this issue 10 months ago • 4 comments

Describe the bug Sidekick fires up without issue, however the first prompt appears to hang, and nothing comes back. Eventually there will be a notification along the lines of "no model responded to request". This was with zero config changes, fresh installation that should point at the local model right? So I downloaded DeepSeek R1 and a couple other models, waited for them to be complete, changed the inference to a freshly downloaded model, and now every prompt is responded to with "Retry" and that's it.

To Reproduce Steps to reproduce the behavior:

  1. Download Sidekick from release page and install
  2. Ask a question, watch app appear to hang (no feedback)
  3. Install new model using the model installer
  4. Ask a question
  5. Only response to anything is "Retry"

Expected behavior I was hoping to play with the RAG capabilities.

Screenshots

Image Image

Desktop (please complete the following information):

  • OS: 14.7.3
  • Hardware: M3 Pro, 36GB RAM, 1TB SSD

Additional context Running Sidekick-Beta 0.0.27 (15)

spacemonkey avatar Mar 22 '25 12:03 spacemonkey

@spacemonkey

Thanks for filing the issue; haven't seen such a catastrophic failure before.

I suspect the 32B model might be too much for 36GB of RAM. Try lowering the context length in Settings -> Inference to lower memory usage, and use the command sudo sysctl iogpu.wired_limit_mb=28672 in the terminal to increase GPU memory.

Sorry for the inconvenience. I'll keep looking into your issue on my end, will post updates here.

johnbean393 avatar Mar 22 '25 13:03 johnbean393

I've tried to fix some of the UI & model loading issues, you can try downloading it here.

johnbean393 avatar Mar 22 '25 14:03 johnbean393

@johnbean393 thanks for getting back so quickly! Downloaded and installed your update.

So I upped GPU, switched to the 7B Qwen model, and set context length to 16384. My understanding is that should have been enough, no? Same response though.

Image

spacemonkey avatar Mar 24 '25 00:03 spacemonkey

I think I might be seeing a similar issue. I downloaded 1.0 rc 6, and while it worked when I downloaded a Qwant model, after closing and re-opening and attempting to connect it to a local LM Studio llama model, now it can't connect to anything. I downloaded the default Qwen 2.5 7B model at first and I'm trying to decide which model to use. I'm thinking I need to get rid of my prefs file for Sidekick and see if that will fix it. Suggestions?

vxbush avatar Apr 16 '25 20:04 vxbush