Initialization for LoRA weights A and B initialized

[sanaullah-06]

System Info

The comment and code are contradictory please anyone explain it to me.

Who can help?

@BenjaminBossan

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder
• My own task or dataset (give details below)

Reproduction

nn.init.zeros_(self.lora_embedding_B[adapter_name])
nn.init.normal_(self.lora_embedding_A[adapter_name])

Expected behavior

nil.

[BenjaminBossan]

You're right, the comment doesn't match the code. After a quick glance at the LoRA paper, I don't see an explicit mention of how LoRA should be initialized for embedding layers. When checking the reference implementation by Microsoft, they, do, however, use the same scheme as we do:

https://github.com/microsoft/LoRA/blob/4c0333854cb905966f8cc4e9a74068c1e507c7b7/loralib/layers.py#L55-L60

Therefore, I think the code is correct.