steampunque

RPC issues and comments

5

Local LAN 1x 1070 1x 4070 1x 4070 configured with new RPC with patched server to use RPC. I did a run to fully offload mixtral Q4_K_M into the 3...

I made a quick patch to server to test RPC running phi-3 fully offloaded onto a remote GPU with the server and all seemed OK, timings: pp: 258.19 tokens per...