Alexander Swerdlow
Alexander Swerdlow
Same problem for me. Very frustrating as I've killed a few training runs by accident at this point trying to kill the right wandb process.
+1 for this feature. Would be super helpful.
Could you provide the script or source you used to generate the annotated BEV images?
Wanted to bump this since I'm looking to use a per-element attention mask. I saw that triton has since removed this as an argument. I'd be happy to help implement...
This would be great to add and I think it's a very necessary feature to allow for resumable training runs. There's no point in using the resume feature if you...
Same issue here. deepspeed: 0.9.2 torch: 2.0.0 lightning: 2.0.2
Of course! I think I should clarify that my initial comment mentions two related but distinct features. The first is just plain variable referencing. For example, I've recently been working...
Sorry for the delay and clever idea! Before I go on, I assume for the 2nd example, you didn't intend to include the `__post_init__`. I ran it myself and it...
Also seeing a symptom of this issue [or at least it seems very related]! I'm seeing a weird occurrence where resuming training causes the master rank to only perform grad...