mphilippnv
mphilippnv
Deepseek v3 doesn't appear to support pipeline parallelism. I get this error when attempting to deploy to 2 8x H100 nodes: ``` NotImplementedError: Pipeline parallelism is only supported for the...
> @july8023 It should work on 4090, generally the models takes about 600GB memory, then you want about 100-300GB for KV cache so feel free to plan around that. @fsaudm...
Just a heads up, I am trying to incorporate this into my pipeline parallel setup for deepseek v2. I modified my docker file to build vllm from this branch. I...
> > @justinthelaw @mphilippnv does this pr solve your problem? @youkaichao I am still having trouble building my docker container with the config changes. I have modified my local vllm...
> @mphilippnv Have you had a chance to verify these changes in your environment? No. I have too much work coming in. Haven't had time to really play with it.
Also seeing this error deploying on H100's using vllm v0.10.2. ``` --port 8002 --model /config/models/model --tensor-parallel-size 8 --disable-log-requests --enable-chunked-prefill --enable-prefix-caching --max-model-len 32768 --served-model-name intern-vl-241b-a28b --trust-remote-code --enable-expert-parallel ```
@Will-So Which EAP release did you install? I installed 0.0.68 and it's not recognizing any of my files when I try using the @ -> files option. This is Pycharm...
@Will-So No I don't see that error anymore. The logs look fine and codebase retrieval seems to work. But I cannot use specific file context anymore. See screenshot. It's always...
Can confirm this is a problem in Pycharm 2024.2.1 with Continue 0.0.64. I had posted in Discord about it but figured I'd post here just for visibility. Thanks!