Muggle666

[BUG] Tensor are not on the same device when enable cpu activation offload

9

I am trying to enable cpu activation offload when training my custom LLAMA model. However, an error occur:  It seems like some inputs of the attention operation is offloaded...