High Initial Loss When Fine-Tuning Gemma Model

[zhangzc21]

Dear authors,

Thans for your great works!

I am currently trying to fine-tune the Gemma-7B model using PiSSA, but I am encountering an issue where the initial loss and grad norm are extremely high.

This doesn't seem to be cuased by the pissa algorithm, since using LoRA to fine-tune Gemma-7B also has similar problem.

Do you have encounted this question, or have any ideas on how to solve it? Thanks a lot!

[Peng-YM]

+1, I have also encountered the same issue when fine-tuning the Qwen-2.5-7B model. The initial loss is approximately an order of magnitude higher than that of the standard LoRA method. For instance, while the standard LoRA achieves an initial loss of around 0.6, PiSSA exhibits an initial loss of approximately 6 or higher.

[gapsong]

@zhangzc21 @Peng-YM did you find some futher answers? I am facing the same problem...

[Peng-YM]

@zhangzc21 @Peng-YM did you find some futher answers? I am facing the same problem...

We ultimately couldn't resolve this issue and ended up abandoning the PiSSA fine-tuning approach

[gapsong]

Me too. Even the answer of them did not help me at all.
I have used the same Hyperparameters as they recommend, but still not working out.
@Peng-YM

[fxmeng]

+1, I have also encountered the same issue when fine-tuning the Qwen-2.5-7B model. The initial loss is approximately an order of magnitude higher than that of the standard LoRA method. For instance, while the standard LoRA achieves an initial loss of around 0.6, PiSSA exhibits an initial loss of approximately 6 or higher.

Hello, we trained Qwen2.5-7B and Gemma-7B using PiSSA with the default hyperparameters, and we did not observe the phenomenon of higher loss. Attached are the training logs.

There may be two possible reasons for the issue you encountered:
1. When performing the PiSSA initialize, you used lora_alpha == lora_r, but during training, you used lora_alpha == 2 * lora_r. In this case,
W != 2 * U[:r] S[:r]V[:r] + U[r:]S[r:]V[r:].
2. Environment-related issues.

If you could share your environment file and the training script, we can further analyze the possible cause of this problem.

trainer_state_metamath_pissa_gemma_7b.json
trainer_state_metamath_pissa_qwen2.5_7b.json