dranger003
commited on
Commit
•
f85aad8
1
Parent(s):
f80614d
Update README.md
Browse files
README.md
CHANGED
@@ -23,7 +23,7 @@ The quants here are meant to test imatrix quantized weights.
|
|
23 |
|
24 |
Quants in this repo are tested running the following command (quants under IQ3 are very sensitive and unreliable so far - the imatrix may require to be trained on FP16 weights rather than Q8_0 and for longer than 200 chunks):
|
25 |
```
|
26 |
-
./build/bin/main -ngl 41 -s 0 -e -p "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nWrite an essay about AI.<|im_end|>\n<|im_start|>assistant\n" -m ggml-dbrx-instruct-16x12b-<<quant-to-test>>.gguf
|
27 |
```
|
28 |
|
29 |
* GGUF importance matrix (imatrix) quants for https://huggingface.co/databricks/dbrx-instruct
|
|
|
23 |
|
24 |
Quants in this repo are tested running the following command (quants under IQ3 are very sensitive and unreliable so far - the imatrix may require to be trained on FP16 weights rather than Q8_0 and for longer than 200 chunks):
|
25 |
```
|
26 |
+
./build/bin/main -ngl 41 -c 4096 -s 0 -e -p "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nWrite an essay about AI.<|im_end|>\n<|im_start|>assistant\n" -m ggml-dbrx-instruct-16x12b-<<quant-to-test>>.gguf
|
27 |
```
|
28 |
|
29 |
* GGUF importance matrix (imatrix) quants for https://huggingface.co/databricks/dbrx-instruct
|