collaborative-attention icon indicating copy to clipboard operation
collaborative-attention copied to clipboard
verified publisher icon epfml

Why is this not used more?

Open EvanKomp opened this issue 8 months ago • 0 comments

Hey all. I am convinced by the paper. I am about to use it for my application and see how it does, but am wondering in your read why this is not more widespread in use compared to normal concatenated MHA? Seems like any big LLM company should be using this for param efficiency. Are they and just not saying so?

EvanKomp avatar May 21 '25 22:05 EvanKomp