sam icon indicating copy to clipboard operation
sam copied to clipboard
verified publisher icon davda54

How to use SAM with torch.cuda.amp.GradScaler

Open ancestor-mithril opened this issue 10 months ago • 2 comments

How can you combine SAM with GradScaler and gradient clipping, because you can't unscale twice.

ancestor-mithril avatar Apr 04 '25 14:04 ancestor-mithril

I am also struggling with that right now. I have tried :

  1. Defining a closure function as global and calling scaler.step(optimizer) -> Fails
  2. Pass closure function as kwarg to scaler.step -> Failed
  3. Double .unscale_ -> Failed

If there is a work around, it would be very useful sharing it with us

savassif avatar Apr 16 '25 08:04 savassif

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] avatar May 07 '25 23:05 stale[bot]