LoRA
LoRA copied to clipboard
[Embedding-bugfix]: reset_parameters
initialize A the same way as the default for nn.Linear and B to zero
But why not use nn.init.kaiming_uniform_(self.lora_A, a=math.sqrt(5)) as class Linear?