LLM-Training-Puzzles

LLM-Training-Puzzles copied to clipboard

Update target time / memory

1

Fixes #6.

EugenHotaj

Target Time/Memory Are Confusing

3

For some of the puzzles, the target memory (and time) are impossible to satisfy. E.g. for puzzle 0, I spent some time trying to get from `2621440` to under the...

EugenHotaj

Hi, this might be a minor thing, but I'm wondering in distributed data parallel, when we aggregate `grad_weights` from all machines using `model.allgather`, since `allgather` performs `sum` operation, shouldn't we...

hzyjerry