Skip to content
forked from artidoro/qlora

ZeQLoRA: Efficient Finetuning of Quantized LLMs with ZeRO and LoRA

License

Notifications You must be signed in to change notification settings

hzg0601/ZeQLoRA

 
 

Repository files navigation

ZeQLoRA: Efficient Finetuning of Quantized LLMs With ZeRO and LoRA

Performance

method num_gpus base_model dataset max_steps train_time train_loss eval_loss
QLoRA 2 huggyllama-7b belle_0.5M 1875 ~37h38m ~ ~
QLoRA 1 huggyllama-7b belle_0.5M 1875 6h43m 1.4971 1.3185
ZeQLoRA 2 huggyllama-7b belle_0.5M 1875 10h45m 1.4682 1.2966

About

ZeQLoRA: Efficient Finetuning of Quantized LLMs with ZeRO and LoRA

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Jupyter Notebook 84.9%
  • Python 11.9%
  • Shell 2.2%
  • HTML 1.0%