DeepEP

DeepEP copied to clipboard

How to use test_low_latency to profile the latency with different batch sizes on different RANKs?

22

As shown in the figure, after assigning different num_tokens parameters to each RANK and running test_low_latency.py, the process gets stuck. What methods can be used to profile the impact of...

nannaer

Where do dispatch and combine need to be synchronized?

1

I'd like to know where **_synchronization across all ranks_** is required for both dispatch and combine operations, using the following code that calls low_latency_dispatch and low_latency_combine as an example. Specifically:...

nannaer

sphish

Support other NVLink scenarios

3

test with 4 rank, 8 rank, 24 rank all pass. code is messy and I will refactor later, and also will try to improve a bit of performance if possible....

fzyzcjy

code is roughly like this; I will work on related things more which will also verify this PR more EDIT: it works well on the target hardware; I will try...

fzyzcjy

Hi teams， I would like to deploy and run DeepEP on an L20 device. However, I noticed that your repository mentions that NVLink-enabled devices are required when using NVSHMEM. The...

Xiaofei-fei

undefined symbol

1

I meet a problem: this build ok: NVSHMEM_DIR=/home/wb/nvshmem python setup.py build #python tests/test_intranode.py Traceback (most recent call last): File "/home/wb/DeepEP/tests/test_intranode.py", line 7, in import deep_ep File "/home/wb/miniconda3/lib/python3.9/site-packages/deep_ep-1.0.0+d5ca449-py3.9-linux-x86_64.egg/deep_ep/__init__.py", line 3, in...

wwj-2017-1117

error: nvshmem API called before nvshmem_init

6

error when run test_low_latency.py: Allocating buffer size: 2115.111296 MB ... /root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init /root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init /root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init /root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem...

guodongrui-gdr

test_low_latency failed

31

I am experiencing an issue with NVSHMEM failing to initialize due to transport errors. The error message indicates that NVSHMEM is unable to detect the system topology and cannot initialize...

hyesung84

crash when set NVSHMEM_IBGDA_ENABLE_MULTI_PORT to 1 in case of dual-port RNIC

1

When I ran test_internode.py in case of dual-port, the environment variable _**NVSHMEM_IBGDA_ENABLE_MULTI_PORT**_ is set to 1. Unfortunately, DeepEP crashed when create rdma team by calling nvshmem_team_split_strided(...) in the following code...

yanminjia