-
|
A couple of days back I found ways to extract Custom training from a model to a small LoRA extract. Kohya_ss provides one such tool. But what I don't understand is what "Network Dimension" means? We have an option to select till 1024. Dimension even influences the file size and I noticed that file size comes to somewhere near the selected dimension. If we select 8 as Dimension, resulting LoRA file will be around and below10MB. If we choose 128, file size will be below 200MB and so-on. Other than this While using the LoRA, selected dimension will be used and there is another value, alpha, which is same as dimension. for example dimension: 8.0, alpha: 8.0 So, My question is, What should be the dimension? which Number produces best result? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
Network dimension refers to rank of the weight adaptation. This parameter dictates entirely the size of the resulting file. The best rank depends on your needs. You could start at some value, and if the results are good, try reducing to save on file size. Some discussion here: #37 Alpha is the scaling applied to the adaptation. The discussion linked also has some examples. The scaling value is independent of rank (it should be a scalar value greater than 0). |
Beta Was this translation helpful? Give feedback.
Network dimension refers to rank of the weight adaptation. This parameter dictates entirely the size of the resulting file. The best rank depends on your needs. You could start at some value, and if the results are good, try reducing to save on file size. Some discussion here: #37
Alpha is the scaling applied to the adaptation. The discussion linked also has some examples. The scaling value is independent of rank (it should be a scalar value greater than 0).