Segmond
Segmond
I'm running alright on P40's, this discussion can be closed.
BTW, I was able to complete the deployment manually. I copied the ymls into the apiserver container and applied them. Did a hello-world and the guestbook deployments. ssh port forwarded...
I'm seeing the same issue, nothing to do with any parameters, it's a bug in the code, with deepseek v3 as well, but Q3_K_XL main: server is listening on http://0.0.0.0:8089...
tool call is no longer optional or a fancy thing to have. An LLM without tool calling is not as useful. I look forward to this.
> This should be fixed now. Use the `-c` command line option when starting `rpc-server` to enable the local cache. Thanks, I'm going to give it a try. I'm curious,...
@iamangus Did you ever figure this out? I'm having the same issue on a similar chasis. I'm using the octominer...
I got this resolved. Just reinstalled the amd driver and installed rocm. Followed the instructions, use the package instruction not the amd install script. Set iommu=pt and I did use...
> > Question: Have you been working on a more up-to-date version of this branch? > > No, I am not working on this and I don't have updates. If...
I'm seeing this bug as well, I'm not passing in --dry-allowed-length 4. main: server is listening on http://0.0.0.0:8089 - starting the main loop srv update_slots: all slots are idle srv...