How to attention weights from esm c

[Jianxinnn]

Hi there, I want return attention weight like esm2 from the latest esm-c, but it seems this feature is unavailable for local usage or through the API server.

[lhallee]

Hello, this is available with my group's ESMC wrapper if you still need this feature. https://huggingface.co/Synthyra/ESMplusplus_small https://huggingface.co/Synthyra/ESMplusplus_large

[Jianxinnn]

Hello, this is available with my group's ESMC wrapper if you still need this feature. https://huggingface.co/Synthyra/ESMplusplus_small https://huggingface.co/Synthyra/ESMplusplus_large

wow, thank you, nice work

[zzhangzzhang]

Thanks for sharing this work! I'm curious if you tried to reproduce the P@L results showed in their report. I would be curious to see if/how they managed to achieve such a big jump in contact prediction.

[lhallee]

Thanks for sharing this work!

I hope it's helpful @zzhangzzhang!

I'm curious if you tried to reproduce the P@L results showed in their report. I would be curious to see if/how they managed to achieve such a big jump in contact prediction.

I haven't tried the P@L before with any pLMs - just not a task my group has been directly interested in. ESM2s contact prediction was based on a learned linear projection of the attention weights. My naive guess behind a jump in performance would be 1.) A stronger base model 2.) Perhaps a convnet instead of a linear layer

If you are interested in collaborating to reproduce this I would love to discuss more. Feel free to email [email protected] if you are interested. Cheers

• Logan