Polisetty V R K Jyothendra Varma
Polisetty V R K Jyothendra Varma
Please review this
Hi @RezaYazdaniAminabadi @mrwyattii @jeffra @awan-10 @cmikeh2 @arashb please review this change.
@shanmugamr1992 can you please let us know how to enable ZeRO 1/2/3 ? Thanks
@shanmugamr1992 please let know how can i enable ZeRO 1/2/3 feature ? raised #1156
> @polisettyvarma, thanks for the modifications and this PR. To ensure we do not break anything, I will test the solution more thoroughly on other cases and update you! @yafshar...
> @polisettyvarma how did you test your implementation? Can you provide an example? @yafshar you mentioned about testing it, i thought you are aware of it. before and after my...
> > > @polisettyvarma how did you test your implementation? Can you provide an example? > > > > > > @yafshar you mentioned about testing it, i thought you...
> @polisettyvarma another point is that, for deepspeed use cases, it already has the options for `bind_cores_to_rank` & `bind_core_list` which is technically covering this PR intention. Unless you plan for...
> @polisettyvarma if there is another point which is not covered by `--bind_cores_to_rank` in DeepSpeed, please state clearly and provide the use-cases, or examples, otherwise please close this PR. Thanks...
> @polisettyvarma what you are doing in this PR `pool_size //= world_size; spawn_context.Pool(pool_size)` is done in a correct way using `numactl` and with core affinity in DeepSpeed `deepspeed --bind_cores_to_rank` ....