Can we get a Q4 without the IMat?
#14
by
yehiaserag
- opened
I was checking for a download and couldn't find a Q4 without an IMat.
Q4 provides the sweet spot between quality and size, so please consider it if possible.
yehiaserag
changed discussion title from
Can we get a Q4 without the IMax?
to Can we get a Q4 without the IMat?
yehiaserag
changed discussion title from
Can we get a Q4 without the IMat?
to Can we get a Q4 without the IMat?
@yehiaserag Hey there, I think we have plenty of Q4's out there, check this link -> https://huggingface.co/models?sort=created&search=gguf+c4ai-command-r-plus
After imatrix came out, Q4 is no longer the best balance between quality and size. q5-k-s will have almost the best quality, and q4-k-s is the new balance point. Other ggufs with 4 are also very close to q4-k-s, so you can freely choose whatever you want based on your memory size.