Welcome to GGUF'n'Go, your go-to tool for easily creating quantized versions of Hugging Face models in the GGUF format. Whether you're a machine learning enthusiast or a professional looking to optimize your models, GGUF'n'Go simplifies the process of converting models into various quantized formats for efficient deployment.
Quantization is a key technique in reducing the size of large language models (LLMs) without significantly compromising their performance. By converting models to lower precision formats, you can save on storage, reduce latency, and enable deployment on resource-constrained environments. GGUF'n'Go supports a wide range of quantization types via Llama.cpp, ensuring flexibility and efficiency for your specific needs.
Ensure that you have curl and unzip installed on your system. These tools are essential for downloading and
extracting the necessary files.
Follow these steps to set up the project:
-
Clone the repository:
git clone git@github.com:thesven/GGUF-n-Go.git cd GGUF-n-Go -
Make the setup script executable and run it. This will download and compile
llama.cpp, as well as download thewiki.train.rawdataset.chmod +x setup.sh ./setup.sh
-
Copy the example configuration file and customize it with the Hugging Face model you wish to convert to the GGUF format:
cp gguf.example.toml gguf.toml
-
Edit
gguf.tomlto specify your model details.
Execute the conversion process with the following command:
python ./gguf_n_go.py --config ./gguf.tomlThis will generate the quantized model in the specified formats.
GGUF'n'Go supports a comprehensive range of quantization types, each offering different trade-offs between model size, performance, and quality. Please see the list of quantization types in the gguf.example.toml file. If you notice any missing, please feel free to add them.
Feel free to fork this repository and submit pull requests to contribute to the project. All ideas and suggestions are welcome!