Toan Nguyen

icon indicating a website link https://tnq177.github.io/

icon location úm ba la xì bùa

Results 1 issues of Toan Nguyen

positional encoding seems different from the paper

In the paper, it says: PE(pos,2i)=sin(pos/10000 ** (2i/dmodel)) PE(pos,2i+1)=cos(pos/10000 ** (2i/dmode)l) So basically for dim i, the denominator should be 10000 ** (2 * (i//2)/dmodel). I rewrite the function as:...