Yuxin Yang
Yuxin Yang
Hi, you can use [this repo](https://github.com/yuxin212/intfold), developed by me, to produce single representations : )
> were you able to solve this error? no, i didn't solve this error
@awgu Thank you for your comment. I am not really sure if they are wrapped in the same FSDP module. You can find the model architecture after FSDP wrapping [here](https://gist.github.com/yuxin212/19c0e5cb0802fe515e743a8fde7c1a51)....
The following is the output from the for loop: ``` lm_head.weight is shared with word_embedding.weight ``` I need some time to rewrite part of the code to put `model.module.get_embeds(q_input_ids)` into...
Okay. I found a solution. I only wrapped the transformer layers, that is `BertLayer` and `BasicTransformerBlock`, instead of the whole model plus the transformer layers. And the error disappeared! Thank...
I am curious if I wrap the `ModuleList` level which contains these transformers, will this still be an issue?
Ahh I see. Does this issue affect performance or efficiency?
If I write a `nn.Module` class with a forward call for these transformer layers, and I wrap this class with FSDP, will this avoid the not all-gather issue?
I found this note: > Attempting to run the forward pass of a submodule that is contained in an FSDP instance is not supported and will result in errors. This...
Thank you so much! I will close this issue for now