README.md

---
license: other
license_name: gemma-terms-of-use
license_link: https://ai.google.dev/gemma/terms
---
GGUF quants for https://huggingface.co/HuggingFaceH4/zephyr-7b-gemma-v0.1  

> Zephyr is a series of language models that are trained to act as helpful assistants. Zephyr 7B Gemma is the third model in the series, and is a fine-tuned version of [google/gemma-7b](https://huggingface.co/google/gemma-7b) that was trained on on a mix of publicly available, synthetic datasets using Direct Preference Optimization (DPO). You can reproduce the training of this model via the recipe provided in the [Alignment Handbook](https://github.com/huggingface/alignment-handbook).

There are few things to consider when using this model:
* Special tokens `<|im_start|>` and `<|im_end|>` are not properly mapped as overrides of `<start_of_turn>` and `<end_of_turn>` (issue in the GGUF)
* Repeat penalty must `1.0` (i.e. disabled) just like with the base model
* The model was not trained with the system instructions (i.e. don't add the `system` instructions part of the chatml template)
* Must stop on special token `<end_of_turn>` instead of `<eos>` otherwise the model goes on forever

Here's a setup that seems to work quite well to chat with the model. The Q4_K is very fast and gives ~90 t/s on a 3090 full offloaded:
```
./main -ins -r "<end_of_turn>" --color -e --in-prefix "<start_of_turn>user\n" --in-suffix "<end_of_turn>\n<start_of_turn>assistant\n" -c 0 --temp 0.7 --repeat-penalty 1.0 -ngl 29 -m ggml-zephyr-7b-gemma-v0.1-q4_k.gguf
```

| Layers | Context | [Template](https://huggingface.co/HuggingFaceH4/zephyr-7b-gemma-v0.1/blob/19186e70e5679c47aaef473ae2fd56e20765088d/tokenizer_config.json#L59) |
| --- | --- | --- |
| <pre>28</pre> | <pre>8192</pre> | <pre>\<\|im_start\|\>user<br>{prompt}\<\|im_end\|\><br>\<\|im_start\|\>assistant<br>{response}</pre> |