Riccardo Arnese

don't know if this may be the issue but if you check the output shape of the sequence head it return you a 64-dimensional matrix of which only the first...

i still think there are some problem with indexing... `client.tokenizer('ACDEFGHIKLMNPQRSTVWY')['input_ids']` outputs: `[0, 5, 23, 13, 9, 18, 6, 21, 12, 15, 4, 20, 17, 14, 16, 10, 8, 11, 7,...

``` seq = ['AAAAAAAAAAAAAAAAA'] with torch.no_grad(): tok = client.tokenizer(seq,add_special_tokens=True,padding=True) ids = torch.tensor( tok['input_ids'],dtype=torch.int64).to('cuda') logit_manual = client(ids).sequence_logits logit_manual[0,1:-1,4:24].argmax(dim=1) + 4 ``` output: ` [20, 5, 5, 5, 5, 5, 5, 5,...

first 3 tokens are special tokens so you consider only the tokens from 4 to 23, then you add 4 to the result to map the output ids back to...

however predicting non-masked tokens is not the way how LLM are intendend to be used. I tried running my code on 1000 random human proteins and using your metric obtaining...

> Hi @j3rk0, my group made a wrapper for this that has full Huggingface integration and batching :) https://huggingface.co/Synthyra/ESMplusplus_small Nice work!!

updating to 3.1.1 solved all previously citated issue, model both work by using tutorial or 'manual' code. however i noticed futher issues. first i compared the output of the model...