README.md · RobertSinclair/README at 0c8b2ec919534fc7cc58816edc8fadcbd1662cf0

metadata

title: README
emoji: 🔥
colorFrom: purple
colorTo: purple
sdk: static
pinned: true

Discord: https://discord.com/channels/@robert_46007

These are my own quantizations (updated almost daily).

The difference with normal quantizations is that I quantize the output and embed tensors to f16.
and the other tensors to 15_k,q6_k or q8_0.
This creates models that are little or not degraded at all and have a smaller size.
They run at about 3-6 t/sec on CPU only using llama.cpp
And obviously faster on computers with potent GPUs

ALL the models were quantized in this way:

python llama.cpp/convert_hf_to_gguf.py --outtype f16 model --outfile model.f16.gguf

quantize.exe --allow-requantize --output-tensor-type f16 --token-embedding-type f16 model.f16.gguf model.f16.q5.gguf q5_k  
quantize.exe --allow-requantize --output-tensor-type f16 --token-embedding-type f16 model.f16.gguf model.f16.q6.gguf q6_k  
quantize.exe --allow-requantize --output-tensor-type f16 --token-embedding-type f16 model.f16.gguf model.f16.q6.gguf q8_0  
quantize.exe --allow-requantize --pure model.f16.gguf model.f16.q8_p.gguf q8_0