Tony comments

Results 6 comments of


                                            Tony

app crashed while installation

I have the same error. Did you fix it?

[QUESTION] Adding new parameters in ColumnParallelLinear/RowParallelLinear raises AssertionError (Communication call has not been issued for this bucket) when using overlap-grad-reduce

Bugs still appear. No idea how to do with it.

test_internode.py runs failed due to "nvshmem detect topo failed"

> Your NIC IP address may not be configured correctly. Thanks! Could you please tell me how to check if the IP address is correct? Both node can access to...

test_internode.py runs failed due to "nvshmem detect topo failed"

> I’m not an expert in this area, but I think you could try using `ib_write_bw` to test connectivity. I tried 'ib_write_bw' on both nodes and I got these: ```log...

test_internode.py runs failed due to "nvshmem detect topo failed"

I solved this problem by seting env: ```shell export NVSHMEM_DISABLE_CUDA_VMM=1 ``` but I don't know why this could work.

test_internode.py runs failed due to "nvshmem detect topo failed"

> > I solved this problem by seting env: > > export NVSHMEM_DISABLE_CUDA_VMM=1 > > but I don't know why this could work. > > Memory fragmentation of NVSHMEM, did...