composable_kernel icon indicating copy to clipboard operation
composable_kernel copied to clipboard
verified publisher icon ROCmSoftwarePlatform

flash_attention forward train

Open guangzlu opened this issue 3 years ago • 0 comments

Added LSE storing into flash attention forward path. Added device random number generator philox. Based on philox, added blockwise dropout. And dropout is applied into flash attention forward path. Flash attention forward training path is finished.

guangzlu avatar Jan 17 '23 02:01 guangzlu