Alex comments

Repositories
Issues
Comments

Results 2 comments of


                                            Alex

Adding additional embeddings after the Q and K are projected, but before the RoPE

Hi, @lucidrains thank you for your answer, basically it is, I am just experimenting with my data and considered also such an option. Could you please advise me or at...

Adding additional embeddings after the Q and K are projected, but before the RoPE

@lucidrains sorry for a late reply, the embeddings I am adding are distinct, not shared across heads.