|
--- |
|
license: apache-2.0 |
|
--- |
|
# What is this? |
|
This is GGUF of [Qwen2-57B-A14B-Instruct](https://huggingface.co/Qwen/Qwen2-57B-A14B-Instruct).<br> |
|
I think this is the world's first successful example to make Qwen2-57B-A14B-Instruct's gguf with imatrix. |
|
|
|
# imatrix dataset |
|
[TFMC/imatrix-dataset-for-japanese-llm](https://huggingface.co/datasets/TFMC/imatrix-dataset-for-japanese-llm).<br> |
|
This dataset contains English and many Japanese sentence. |
|
|
|
# How to made it |
|
First I made Q6_K then tried to convert it to i-quants with ``--allow-requantize`` option.<br> |
|
Surprisingly (re)quantizeing process has been completed without ploblems.<br> |
|
I will show more detail below. |
|
|
|
### Step1: Making GGUF of f16 |
|
At first, I converted safetensors to GGUF. |
|
|
|
### Step2: Converting f16 to Q8_0 |
|
Second, I converted f16 to Q8_0.<br> |
|
This is aiming to accelerate next process because I don't have enough memory to deal with high precision tensor. |
|
|
|
### Step3: Calculating imatrix |
|
I calculate imatrix with the Q8_0.<br> |
|
I seem to some people succeed to calculate imatrix so I think anyone can make imatrix.<br> |
|
At this time, I use '-fa' option because I want to finish culculation as much as possible.<br> |
|
However, later I knew some people claim Qwen2 needs `-fa` option to work correctly. |
|
|
|
### Step4: Making Q6_K temporarily |
|
This is most important step. Firstly I converted f16 to Q6_K.<br> |
|
Never try to make i-quants directly. No one may succeed to make it directly. |
|
|
|
### Step5: Converting Q6 to i-quants with the imatrix |
|
I converted the Q6_K to i-quants with imatrix.<br> |
|
Strangely the process has been finished and the i-quants may work. |
|
|
|
### environment |
|
GeForce RTX 3090 and llama.cpp windows binary b3065 |
|
|
|
# License |
|
Apache 2.0 |
|
|
|
# Developer |
|
Alibaba Cloud |
