Is it possible to restart finetuning from an existing checkpoint?

[chaovven]

🚀 The feature, motivation and pitch

Hi, I'm curious if I can resume the fine-tuning process from an existing checkpoint?

Right now, I have to start fine-tuning from scratch. Being able to pick up where I left off from an existing checkpoint would be incredibly helpful. Is there a way to achieve this in llama-recipes? I'm using FSDP+PEFT for finetuning. Thanks a lot!

Motivation:
The fine-tuning process stops after specified epochs (e.g., 3 epochs), but later I may find that 3 epochs are not enough and I want to fine-tune for more epochs. Currently, I would have to restart the fine-tuning from scratch. But if I could resume the fine-tuning from the existing checkpoint for epoch 3, then I can save time and resources.

Alternatives

No response

Additional context

No response

[HamidShojanazeri]

Thanks @chaovven for the feature request, totally agreed, wonder if you are interested to send a PR? I can help to merge it.

[wukaixingxp]

Hi! This PR has been merged. Hopefully it can solve your issue. Please give it a try and let me know if there is any problem.